Sep 17 23:10:39.436843 ip-10-0-134-66 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:10:39.841014 ip-10-0-134-66 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:39.841014 ip-10-0-134-66 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:10:39.841014 ip-10-0-134-66 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:39.841014 ip-10-0-134-66 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:10:39.841014 ip-10-0-134-66 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:39.841507 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.841063    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:10:39.846734 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846701    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.846734 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846732    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846743    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846750    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846755    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846760    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846765    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846771    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846778    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846784    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846788    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846793    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846798    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846803    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846813    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846817    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846821    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846826    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846830    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846834    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.846859 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846838    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846843    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846855    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846861    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846867    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846872    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846882    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846887    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846892    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846896    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846900    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846905    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846909    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846914    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846918    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846922    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846926    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846933    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846941    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.847646 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846945    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846949    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846954    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846958    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846962    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846966    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846970    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846974    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846978    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846983    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.846989    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847000    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847004    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847008    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847013    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847019    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847025    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847028    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847031    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847034    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.848182 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847037    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847039    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847042    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847045    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847050    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847053    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847056    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847059    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847062    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847065    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847067    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847070    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847073    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847075    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847080    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847083    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847088    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847091    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847093    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847096    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.848988 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847099    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847102    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847104    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847107    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847110    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847112    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847115    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847119    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847124    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847127    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847130    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847133    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847136    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847139    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847142    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847144    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847147    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847150    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847152    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.847155    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.849838 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.848999    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849013    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849020    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849024    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849028    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849033    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849037    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849042    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849046    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849050    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849054    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849058    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849062    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849066    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849070    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849075    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849079    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849084    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849088    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849092    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.850708 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849096    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849101    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849105    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849109    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849113    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849120    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849127    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849131    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849136    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849140    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849144    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849149    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849153    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849158    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849163    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849167    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849171    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849175    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849180    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.851639 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849184    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849188    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849192    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849196    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849201    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849204    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849209    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849213    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849217    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849221    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849226    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849230    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849234    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849238    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849242    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849246    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849251    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849255    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849259    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849263    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.852313 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849267    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849271    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849276    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849280    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849286    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849290    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849297    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849303    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849309    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849314    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849319    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849324    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849329    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849334    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849338    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849343    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849347    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849351    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849355    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.852841 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849359    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849364    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849368    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849372    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849376    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849384    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849389    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849393    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849397    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849402    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849406    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849410    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849414    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849425    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849430    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849435    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849439    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849443    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849448    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849453    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.853362 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.849457    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850053    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850062    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850066    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850071    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850074    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850079    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850083    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850088    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850092    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850096    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850100    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850105    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850109    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850113    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850117    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850121    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850130    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850136    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.853898 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850141    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850145    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850149    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850154    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850158    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850162    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850166    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850171    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850175    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850179    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850184    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850190    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850195    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850199    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850203    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850208    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850212    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850217    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850221    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850225    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.854509 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850230    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850234    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850238    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850242    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850246    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850251    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850255    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850259    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850264    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850268    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850272    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850277    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850281    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850286    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850290    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850294    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850299    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850303    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850307    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850311    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.855158 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850315    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850319    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850323    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850331    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850335    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850340    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850344    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850349    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850354    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850358    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850362    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850366    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850373    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850378    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850383    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850387    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850392    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850398    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850403    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.855674 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850408    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850412    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850417    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850421    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850425    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850430    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850434    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850438    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850442    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850446    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850450    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850454    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850458    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850462    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850466    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850470    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850477    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850481    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850485    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850489    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.856215 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850493    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.850497    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850614    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850665    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850675    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850681    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850688    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850693    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850699    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850705    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850710    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850714    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850719    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850732    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850737    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850742    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850746    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850751    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850757    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850762    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850768    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850772    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850777    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850782    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:10:39.856760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850788    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850793    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850798    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850803    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850810    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850815    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850819    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850824    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850831    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850836    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850842    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850847    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850852    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850856    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850861    2566 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850866    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850874    2566 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850879    2566 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850884    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850888    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850893    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850900    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850904    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850909    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850914    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:10:39.857353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850918    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850923    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850929    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850934    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850938    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850943    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850947    2566 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850953    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850958    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850963    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850968    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850975    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850979    2566 flags.go:64] FLAG: --help="false"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850984    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-134-66.ec2.internal"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850989    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850994    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.850998    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851004    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851009    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851014    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851019    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851023    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851028    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:10:39.857972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851033    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851038    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851042    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851047    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851051    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851056    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851061    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851066    2566 flags.go:64] FLAG: --lock-file=""
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851070    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851075    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851079    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851087    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851093    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851098    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851102    2566 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851107    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851112    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851117    2566 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851121    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851127    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851136    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851142    2566 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851147    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851151    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851156    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:10:39.858626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851160    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851165    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851170    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851174    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851185    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851190    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851195    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851199    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851204    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851211    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851216    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851221    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851225    2566 flags.go:64] FLAG: --port="10250"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851230    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851234    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0639544c400d66f80"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851240    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851244    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851249    2566 flags.go:64] FLAG: --register-node="true"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851254    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851259    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851265    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851270    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851274    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851280    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851285    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:10:39.859252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851290    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851295    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851302    2566 flags.go:64] FLAG: --runonce="false"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851308    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851313    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851317    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851322    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851327    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851332    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851336    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851341    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851345    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851350    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851355    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851359    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851364    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851369    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851373    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851381    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851386    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851390    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851397    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851401    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851406    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851410    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:10:39.859889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851415    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851420    2566 flags.go:64] FLAG: --v="2"
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851426    2566 flags.go:64] FLAG: --version="false"
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851433    2566 flags.go:64] FLAG: --vmodule=""
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851439    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.851444    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851630    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851638    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851642    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851649    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851655    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851659    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851664    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851668    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851672    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851676    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851680    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851684    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851689    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851693    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851698    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.860520 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851702    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851706    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851711    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851716    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851720    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851724    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851728    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851733    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851737    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851741    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851745    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851749    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851753    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851757    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851761    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851767    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851771    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851776    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851780    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851785    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.861041 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851790    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851795    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851799    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851803    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851807    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851812    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851816    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851820    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851824    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851828    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851832    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851836    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851840    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851844    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851848    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851852    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851856    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851860    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851864    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851869    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.861510 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851873    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851877    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851881    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851885    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851892    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851899    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851906    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851913    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851919    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851923    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851928    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851932    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851939    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851944    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851949    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851953    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851958    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851962    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851966    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.862021 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851970    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851975    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851979    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851983    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851988    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851992    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.851996    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852000    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852004    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852009    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852013    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852017    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852021    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852026    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852030    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852034    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852038    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852042    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852046    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852050    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.862496 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852057    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852063    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852070    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852075    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.852079    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.852807    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.859982    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.859997    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.860899    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.860908    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860961    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860966    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860970    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860973    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860976    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.863008 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860979    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860982    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860985    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860988    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860991    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860995    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.860997    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861002    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861006    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861009    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861012    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861015    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861018    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861021    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861023    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861026    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861029    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861031    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861034    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861037    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.863370 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861039    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861042    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861045    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861048    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861052    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861055    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861058    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861061    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861063    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861066    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861069    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861071    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861074    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861077    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861080    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861082    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861085    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861087    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861090    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861093    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.863983 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861096    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861098    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861101    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861104    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861107    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861109    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861112    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861114    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861117    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861120    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861122    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861126    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861131    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861134    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861136    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861139    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861142    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861145    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861148    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.864472 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861150    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861153    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861156    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861158    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861161    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861164    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861166    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861169    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861172    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861174    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861177    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861179    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861182    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861185    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861187    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861190    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861193    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861196    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861198    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861201    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.864947 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861203    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861206    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861209    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861212    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861215    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861218    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861220    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861223    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861226    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861228    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861231    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861236    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861239    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861243    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.865434 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861245    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.861251    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861360    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861365    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861368    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861371    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861374    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861377    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861380    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861382    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861385    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861387    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861390    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861393    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861395    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.865794 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861398    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861400    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861403    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861406    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861408    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861411    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861414    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861416    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861419    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861422    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861425    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861427    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861430    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861432    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861435    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861438    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861440    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861443    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861445    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861448    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.866170 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861450    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861453    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861455    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861458    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861461    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861463    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861466    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861468    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861471    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861473    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861476    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861479    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861481    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861484    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861486    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861489    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861491    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861494    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861498    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.867273 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861501    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861505    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861508    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861511    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861514    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861516    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861520    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861522    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861525    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861528    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861531    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861533    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861536    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861538    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861541    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861543    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861546    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861549    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861551    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.867740 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861554    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861557    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861560    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861562    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861565    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861567    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861570    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861593    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861597    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861599    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861602    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861604    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861607    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861610    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861613    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861616    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861618    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861621    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861623    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.868205 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861627    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861631    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861636    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861639    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861643    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861646    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861649    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861652    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:39.861655    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.861660    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.861755    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.864115    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.865119    2566 server.go:1019] "Starting client certificate rotation"
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.865215    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:39.868683 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.865253    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:39.869025 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.867827    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:39.873348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.873334    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:39.885156 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.885139    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:10:39.886922 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.886902    2566 log.go:25] "Validated CRI v1 image API"
Sep 17 23:10:39.887711 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.887695    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:10:39.895326 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.895311    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:39.895916 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.895900    2566 fs.go:135] Filesystem UUIDs: map[47cb05e2-d0f2-4f89-8fd7-af7753c6cc11:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a5c41bc4-1315-4165-be2f-30b3bdf0902f:/dev/nvme0n1p4]
Sep 17 23:10:39.895964 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.895916    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:10:39.900870 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.900773    2566 manager.go:216] Machine: {Timestamp:2026-09-17 23:10:39.899349267 +0000 UTC m=+0.351180747 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100044 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c423c6e93a8872ec86b7a0aec5edf SystemUUID:ec2c423c-6e93-a887-2ec8-6b7a0aec5edf BootID:a21e70d3-edb9-4edc-96c5-f4f74ea6ab2f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:9c:e9:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:9c:e9:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:dd:16:3a:5b:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:10:39.900870 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.900869    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:10:39.900974 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.900957    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:10:39.904969 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.904948    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:10:39.905090 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.904972    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:10:39.905822 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.905811    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:10:39.905860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.905824    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:10:39.905860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.905831    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:39.906626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.906616    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:39.906686 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.906678    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:10:39.911554 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.911543    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:39.911687 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.911677    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:10:39.911732 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.911706    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:10:39.911732 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.911715    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:10:39.911732 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.911727    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:10:39.911805 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.911736    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:10:39.911805 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.911745    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:10:39.912041 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.912030    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:39.912087 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.912048    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:39.914254 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.914237    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:10:39.917299 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.917284    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:10:39.917356 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.917316    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:10:39.918890 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918877    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:10:39.918946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918899    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:10:39.918946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918908    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:10:39.918946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918917    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:10:39.918946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918929    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:10:39.918946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918938    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:10:39.919073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918962    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:10:39.919073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918971    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:10:39.919073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918978    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:10:39.919073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918984    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:10:39.919073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.918995    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:10:39.919073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.919001    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:10:39.919821 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.919804    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:10:39.919821 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.919821    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:10:39.921963 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.921921    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:10:39.922063 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.922041    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:10:39.923518 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.923504    2566 server.go:1268] "Started kubelet"
Sep 17 23:10:39.923611 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.923533    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:10:39.928370 ip-10-0-134-66 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:10:39.929777 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.923623    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:10:39.929861 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.929850    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:10:39.930876 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.930856    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:10:39.933146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.932292    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:10:39.934908 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.934876    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:10:39.935018 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.935001    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:10:39.936220 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.935056    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-66.ec2.internal.18d63e844fe668f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-66.ec2.internal,UID:ip-10-0-134-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-66.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.923456249 +0000 UTC m=+0.375287729,LastTimestamp:2026-09-17 23:10:39.923456249 +0000 UTC m=+0.375287729,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-66.ec2.internal,}"
Sep 17 23:10:39.940848 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.940782    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:39.941412 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.941397    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:10:39.941546 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.941526    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:39.942397 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942146    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:10:39.942397 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942173    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:10:39.942397 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942184    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:10:39.942397 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942326    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:10:39.942397 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942334    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:10:39.942397 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942386    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nm9dp"
Sep 17 23:10:39.942664 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942599    2566 factory.go:55] Registering systemd factory
Sep 17 23:10:39.942664 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942612    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:10:39.942844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942831    2566 factory.go:153] Registering CRI-O factory
Sep 17 23:10:39.942896 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942846    2566 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:10:39.942896 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942895    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:10:39.942958 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942908    2566 factory.go:103] Registering Raw factory
Sep 17 23:10:39.942958 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.942917    2566 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:10:39.943268 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.943251    2566 manager.go:349] Starting recovery of all containers
Sep 17 23:10:39.947061 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.947029    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:10:39.947311 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.947285    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:10:39.953910 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.953891    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nm9dp"
Sep 17 23:10:39.954187 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.954167    2566 manager.go:354] Recovery completed
Sep 17 23:10:39.958017 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.958004    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.959834 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.959817    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.959894 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.959846    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.959894 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.959856    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.960407 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.960393    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:10:39.960407 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.960406    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:10:39.960534 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.960416    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.960534 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.960419    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:39.961466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.961450    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.961520 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.961492    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.961520 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:39.961444    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-66.ec2.internal.18d63e8452117cfa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-66.ec2.internal,UID:ip-10-0-134-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-66.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.95983385 +0000 UTC m=+0.411665329,LastTimestamp:2026-09-17 23:10:39.95983385 +0000 UTC m=+0.411665329,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-66.ec2.internal,}"
Sep 17 23:10:39.961627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.961508    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.962505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.962489    2566 policy_none.go:49] "None policy: Start"
Sep 17 23:10:39.962568 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.962509    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:10:39.962568 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.962523    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:10:39.964625 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:39.964612    2566 policy_none.go:47] "Start"
Sep 17 23:10:40.002351 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.002248    2566 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.002373    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.002401    2566 server.go:87] "Starting device plugin registration server"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.002622    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.002634    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.002742    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.002826    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.002836    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.003242    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.003282    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.005647    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.007686    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.007726    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.007742    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.007808    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:10:40.018678 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.009489    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:40.103519 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.103472    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:40.104270 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.104254    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:40.104349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.104278    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:40.104349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.104291    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:40.104349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.104314    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.108403 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.108388    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal"]
Sep 17 23:10:40.108449 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.108441    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:40.109894 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.109878    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:40.109963 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.109903    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:40.109963 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.109912    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:40.112152 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112139    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:40.112272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.112357 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112291    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:40.112757 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112741    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:40.112836 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112762    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:40.112836 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112772    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:40.112836 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112799    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:40.112836 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112821    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:40.112836 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.112836    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:40.115302 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.115288    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.115369 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.115308    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-66.ec2.internal\": node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.115369 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.115346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.115369 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.115367    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:40.116042 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.116017    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:40.116128 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.116052    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:40.116128 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.116063    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:40.136052 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.136028    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.138117 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.138104    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-66.ec2.internal\" not found" node="ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.142390 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.142372    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-66.ec2.internal\" not found" node="ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.142497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.142482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52fd5495c101430ae182aa561a48cae8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal\" (UID: \"52fd5495c101430ae182aa561a48cae8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.142542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.142506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52fd5495c101430ae182aa561a48cae8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal\" (UID: \"52fd5495c101430ae182aa561a48cae8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.142607 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.142537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b015008b02def3f3ba52667fd2a6cf91-config\") pod \"kube-apiserver-proxy-ip-10-0-134-66.ec2.internal\" (UID: \"b015008b02def3f3ba52667fd2a6cf91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.236094 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.236063    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.243333 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.243312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52fd5495c101430ae182aa561a48cae8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal\" (UID: \"52fd5495c101430ae182aa561a48cae8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.243410 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.243344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52fd5495c101430ae182aa561a48cae8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal\" (UID: \"52fd5495c101430ae182aa561a48cae8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.243410 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.243378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b015008b02def3f3ba52667fd2a6cf91-config\") pod \"kube-apiserver-proxy-ip-10-0-134-66.ec2.internal\" (UID: \"b015008b02def3f3ba52667fd2a6cf91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.243505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.243421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52fd5495c101430ae182aa561a48cae8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal\" (UID: \"52fd5495c101430ae182aa561a48cae8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.243505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.243439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52fd5495c101430ae182aa561a48cae8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal\" (UID: \"52fd5495c101430ae182aa561a48cae8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.243505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.243499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b015008b02def3f3ba52667fd2a6cf91-config\") pod \"kube-apiserver-proxy-ip-10-0-134-66.ec2.internal\" (UID: \"b015008b02def3f3ba52667fd2a6cf91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.336496 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.336467    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.437110 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.437057    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.443296 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.443281    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.447541 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.447521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:40.537661 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.537637    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.638199 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.638173    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.738695 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.738645    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.839197 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.839174    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.865679 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.865661    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:10:40.866173 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.865819    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:40.939713 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:40.939693    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:40.960473 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.960444    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:05:39 +0000 UTC" deadline="2028-06-23 12:24:02.505816504 +0000 UTC"
Sep 17 23:10:40.960550 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:40.960474    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15469h13m21.545346103s"
Sep 17 23:10:41.040659 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.040617    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:41.092233 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:41.092207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb015008b02def3f3ba52667fd2a6cf91.slice/crio-234d5fee7df627aa45b1524043bf37394c329f4c4c26ebc0f2143a35527f347d WatchSource:0}: Error finding container 234d5fee7df627aa45b1524043bf37394c329f4c4c26ebc0f2143a35527f347d: Status 404 returned error can't find the container with id 234d5fee7df627aa45b1524043bf37394c329f4c4c26ebc0f2143a35527f347d
Sep 17 23:10:41.092398 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:41.092386    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fd5495c101430ae182aa561a48cae8.slice/crio-564af9949f8a82a121c19e84fb95bab2a0802c70270b276153dc7e4676021c66 WatchSource:0}: Error finding container 564af9949f8a82a121c19e84fb95bab2a0802c70270b276153dc7e4676021c66: Status 404 returned error can't find the container with id 564af9949f8a82a121c19e84fb95bab2a0802c70270b276153dc7e4676021c66
Sep 17 23:10:41.096835 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.096823    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:10:41.141135 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.141114    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:41.241407 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.241386    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:41.340453 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.340393    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:41.341678 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.341662    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:41.441750 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.441724    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-66.ec2.internal\" not found"
Sep 17 23:10:41.501326 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.501307    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:41.507673 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.507656    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:41.542671 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.542649    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:41.552453 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.552430    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:41.553660 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.553638    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal"
Sep 17 23:10:41.561205 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.561183    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:41.912020 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.911997    2566 apiserver.go:52] "Watching apiserver"
Sep 17 23:10:41.918933 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.918911    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:10:41.921406 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.921376    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-kmt2w","openshift-network-operator/iptables-alerter-9h7gv","openshift-ovn-kubernetes/ovnkube-node-x6rgw","kube-system/global-pull-secret-syncer-jkmmb","kube-system/konnectivity-agent-qhsgn","kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5","openshift-cluster-node-tuning-operator/tuned-tbgm7","openshift-dns/node-resolver-gq5q5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal","openshift-multus/multus-additional-cni-plugins-dvw78","openshift-image-registry/node-ca-nvz6b","openshift-multus/multus-7zhxm","openshift-multus/network-metrics-daemon-zd5lf"]
Sep 17 23:10:41.923684 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.923659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:41.925846 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.925827    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:41.926435 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.926231    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.926435 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.926252    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ktmvr\""
Sep 17 23:10:41.926435 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.926301    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.927853 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.927821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.928162 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.928138    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:10:41.928286 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.928256    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.928376 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.928362    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.928433 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.928384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nxstj\""
Sep 17 23:10:41.929874 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.929859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:41.929962 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.929918    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:41.930483 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.930466    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:10:41.930784 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.930767    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jsf4s\""
Sep 17 23:10:41.931047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.931023    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.931209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.931197    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.931801 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.931784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:41.933889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.933868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:41.933978 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.933925    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:41.933978 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.933944    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:10:41.934217 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.934205    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:10:41.934374 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.934360    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sj2vg\""
Sep 17 23:10:41.936149 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.935871    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.937955 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.937936    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.938299 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.938281    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.938424 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.938403    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhl8z\""
Sep 17 23:10:41.938494 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.938481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.940505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.940488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.941170 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.941147    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:10:41.941170 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.941162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zc246\""
Sep 17 23:10:41.941254 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.941178    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:10:41.941415 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.941397    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.942146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.941691    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.942146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.941881    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:41.942456 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.942335    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:10:41.942456 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.942384    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:10:41.942696 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.942681    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:10:41.942764 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.942708    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:10:41.942814 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.942784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:41.942946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.942928    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.943095 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.943083    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:10:41.943434 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.943416    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t9t5p\""
Sep 17 23:10:41.943502 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.943475    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.945462 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.945084    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.945462 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.945261    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.945462 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.945442    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:10:41.945684 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.945483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n4jtd\""
Sep 17 23:10:41.946274 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.946258    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:41.948609 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.948594    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f58g8\""
Sep 17 23:10:41.948743 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.948668    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:10:41.948985 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.948970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:41.949164 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:41.949133    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:41.952876 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.952862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-lib-modules\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.952965 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.952885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwfjc\" (UniqueName: \"kubernetes.io/projected/fab75934-578e-42db-963e-c4eadf56b841-kube-api-access-vwfjc\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.952965 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.952902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.952965 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.952923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.952965 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.952960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-system-cni-dir\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.952989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysconfig\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysctl-conf\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/938d9893-bdce-4bd2-8898-0ab74c7fc325-tmp-dir\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysctl-d\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0be814cf-7b90-4d33-9f35-f73fdf631402-serviceca\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-kubelet\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-slash\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-ovn\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0be814cf-7b90-4d33-9f35-f73fdf631402-host\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-var-lib-kubelet\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-cni-netd\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3162a2e0-39fc-43da-97f6-18fd0932c183-konnectivity-ca\") pod \"konnectivity-agent-qhsgn\" (UID: \"3162a2e0-39fc-43da-97f6-18fd0932c183\") " pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-systemd\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab75934-578e-42db-963e-c4eadf56b841-tmp\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.953488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tsr\" (UniqueName: \"kubernetes.io/projected/fa5df00c-41f7-40b6-973d-c7090ae9dbca-kube-api-access-s7tsr\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-run-netns\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-log-socket\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-env-overrides\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944hk\" (UniqueName: \"kubernetes.io/projected/938d9893-bdce-4bd2-8898-0ab74c7fc325-kube-api-access-944hk\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-dbus\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vt6\" (UniqueName: \"kubernetes.io/projected/2199a550-7b6f-455c-8731-0aedbffba902-kube-api-access-n2vt6\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cnibin\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-modprobe-d\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.953946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-host\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.953981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fab75934-578e-42db-963e-c4eadf56b841-etc-tuned\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-cni-bin\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-device-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2zwk\" (UniqueName: \"kubernetes.io/projected/f4e50a1d-15b6-42be-bcd4-641f72602ad1-kube-api-access-b2zwk\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2199a550-7b6f-455c-8731-0aedbffba902-iptables-alerter-script\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-systemd\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-node-log\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gz4\" (UniqueName: \"kubernetes.io/projected/0be814cf-7b90-4d33-9f35-f73fdf631402-kube-api-access-r4gz4\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-kubelet-config\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954250    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2199a550-7b6f-455c-8731-0aedbffba902-host-slash\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/938d9893-bdce-4bd2-8898-0ab74c7fc325-hosts-file\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-systemd-units\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-var-lib-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.954543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovnkube-script-lib\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3162a2e0-39fc-43da-97f6-18fd0932c183-agent-certs\") pod \"konnectivity-agent-qhsgn\" (UID: \"3162a2e0-39fc-43da-97f6-18fd0932c183\") " pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954402    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-run\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-sys\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-etc-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovnkube-config\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-kubernetes\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.955171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.954677    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgzpj\" (UniqueName: \"kubernetes.io/projected/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-kube-api-access-zgzpj\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:41.996262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:41.996240    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:42.012066 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.012020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal" event={"ID":"52fd5495c101430ae182aa561a48cae8","Type":"ContainerStarted","Data":"564af9949f8a82a121c19e84fb95bab2a0802c70270b276153dc7e4676021c66"}
Sep 17 23:10:42.014142 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.014114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal" event={"ID":"b015008b02def3f3ba52667fd2a6cf91","Type":"ContainerStarted","Data":"234d5fee7df627aa45b1524043bf37394c329f4c4c26ebc0f2143a35527f347d"}
Sep 17 23:10:42.014441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.014425    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d2zn5"
Sep 17 23:10:42.020718 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.020688    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d2zn5"
Sep 17 23:10:42.043354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.043339    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:10:42.054999 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.054983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:42.055076 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-k8s-cni-cncf-io\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.055076 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxv4\" (UniqueName: \"kubernetes.io/projected/2f99df5d-824d-4474-a81b-7cda0839ae43-kube-api-access-ksxv4\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.055154 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-systemd-units\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055154 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-var-lib-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055154 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovnkube-script-lib\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-var-lib-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3162a2e0-39fc-43da-97f6-18fd0932c183-agent-certs\") pod \"konnectivity-agent-qhsgn\" (UID: \"3162a2e0-39fc-43da-97f6-18fd0932c183\") " pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:42.055291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-systemd-units\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.055291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-run\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-sys\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-etc-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovnkube-config\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-kubernetes\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-sys\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzpj\" (UniqueName: \"kubernetes.io/projected/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-kube-api-access-zgzpj\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-multus-certs\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-lib-modules\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwfjc\" (UniqueName: \"kubernetes.io/projected/fab75934-578e-42db-963e-c4eadf56b841-kube-api-access-vwfjc\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-system-cni-dir\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-hostroot\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-conf-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysconfig\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovnkube-script-lib\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysctl-conf\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-etc-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/938d9893-bdce-4bd2-8898-0ab74c7fc325-tmp-dir\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055770    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f99df5d-824d-4474-a81b-7cda0839ae43-cni-binary-copy\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-system-cni-dir\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.055921 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.055957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-lib-modules\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-run\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysconfig\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysctl-conf\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-socket-dir-parent\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-kubernetes\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysctl-d\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056161    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/938d9893-bdce-4bd2-8898-0ab74c7fc325-tmp-dir\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0be814cf-7b90-4d33-9f35-f73fdf631402-serviceca\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovnkube-config\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-cni-multus\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-kubelet\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-sysctl-d\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-slash\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-ovn\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.056642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-slash\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-kubelet\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0be814cf-7b90-4d33-9f35-f73fdf631402-host\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0be814cf-7b90-4d33-9f35-f73fdf631402-host\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-system-cni-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-var-lib-kubelet\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-cni-netd\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.057250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0be814cf-7b90-4d33-9f35-f73fdf631402-serviceca\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:42.057633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.057633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.056421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-ovn\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.057755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4e50a1d-15b6-42be-bcd4-641f72602ad1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.057755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.057893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3162a2e0-39fc-43da-97f6-18fd0932c183-konnectivity-ca\") pod \"konnectivity-agent-qhsgn\" (UID: \"3162a2e0-39fc-43da-97f6-18fd0932c183\") " pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:42.057893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.057893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsnvg\" (UniqueName: \"kubernetes.io/projected/bdb8ea08-57db-405c-ac91-9de219efe014-kube-api-access-wsnvg\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:42.057893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-var-lib-kubelet\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.057893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-cni-netd\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.057893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-cni-bin\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-systemd\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab75934-578e-42db-963e-c4eadf56b841-tmp\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.057991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tsr\" (UniqueName: \"kubernetes.io/projected/fa5df00c-41f7-40b6-973d-c7090ae9dbca-kube-api-access-s7tsr\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-kubelet\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-run-netns\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.058151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-log-socket\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.058627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058172    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-run-netns\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.058627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-env-overrides\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.058627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3162a2e0-39fc-43da-97f6-18fd0932c183-konnectivity-ca\") pod \"konnectivity-agent-qhsgn\" (UID: \"3162a2e0-39fc-43da-97f6-18fd0932c183\") " pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:42.058627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-log-socket\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.058627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-env-overrides\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.058639    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-systemd\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.058716    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret podName:50534f9a-9cd4-4dfb-9e76-02e8f2247ed7 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:42.558686977 +0000 UTC m=+3.010518463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret") pod "global-pull-secret-syncer-jkmmb" (UID: "50534f9a-9cd4-4dfb-9e76-02e8f2247ed7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944hk\" (UniqueName: \"kubernetes.io/projected/938d9893-bdce-4bd2-8898-0ab74c7fc325-kube-api-access-944hk\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-dbus\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-etc-kubernetes\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.058851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vt6\" (UniqueName: \"kubernetes.io/projected/2199a550-7b6f-455c-8731-0aedbffba902-kube-api-access-n2vt6\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-dbus\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.058992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cnibin\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-modprobe-d\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e50a1d-15b6-42be-bcd4-641f72602ad1-cnibin\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-etc-modprobe-d\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.059209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-host\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fab75934-578e-42db-963e-c4eadf56b841-etc-tuned\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-cni-bin\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fab75934-578e-42db-963e-c4eadf56b841-host\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-cni-bin\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-device-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2zwk\" (UniqueName: \"kubernetes.io/projected/f4e50a1d-15b6-42be-bcd4-641f72602ad1-kube-api-access-b2zwk\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-netns\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2199a550-7b6f-455c-8731-0aedbffba902-iptables-alerter-script\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-systemd\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-node-log\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gz4\" (UniqueName: \"kubernetes.io/projected/0be814cf-7b90-4d33-9f35-f73fdf631402-kube-api-access-r4gz4\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-kubelet-config\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:42.059681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-kubelet-config\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-systemd\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-run-openvswitch\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.059814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-node-log\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2199a550-7b6f-455c-8731-0aedbffba902-iptables-alerter-script\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060117    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-cnibin\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-daemon-config\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-cni-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2199a550-7b6f-455c-8731-0aedbffba902-host-slash\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:42.060262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa5df00c-41f7-40b6-973d-c7090ae9dbca-device-dir\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.060700 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/938d9893-bdce-4bd2-8898-0ab74c7fc325-hosts-file\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:42.060700 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/938d9893-bdce-4bd2-8898-0ab74c7fc325-hosts-file\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:42.060700 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.060700 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.060459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2199a550-7b6f-455c-8731-0aedbffba902-host-slash\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:42.062123 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.061260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.062123 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.061315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fab75934-578e-42db-963e-c4eadf56b841-etc-tuned\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.062123 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.061610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3162a2e0-39fc-43da-97f6-18fd0932c183-agent-certs\") pod \"konnectivity-agent-qhsgn\" (UID: \"3162a2e0-39fc-43da-97f6-18fd0932c183\") " pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:42.063878 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.063858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab75934-578e-42db-963e-c4eadf56b841-tmp\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.064462 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.064447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzpj\" (UniqueName: \"kubernetes.io/projected/e18d7e89-9ce2-4526-a68e-00f4ef8e82fe-kube-api-access-zgzpj\") pod \"ovnkube-node-x6rgw\" (UID: \"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.064638 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.064589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwfjc\" (UniqueName: \"kubernetes.io/projected/fab75934-578e-42db-963e-c4eadf56b841-kube-api-access-vwfjc\") pod \"tuned-tbgm7\" (UID: \"fab75934-578e-42db-963e-c4eadf56b841\") " pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.068860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.068840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-944hk\" (UniqueName: \"kubernetes.io/projected/938d9893-bdce-4bd2-8898-0ab74c7fc325-kube-api-access-944hk\") pod \"node-resolver-gq5q5\" (UID: \"938d9893-bdce-4bd2-8898-0ab74c7fc325\") " pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:42.068946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.068890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tsr\" (UniqueName: \"kubernetes.io/projected/fa5df00c-41f7-40b6-973d-c7090ae9dbca-kube-api-access-s7tsr\") pod \"aws-ebs-csi-driver-node-6qdd5\" (UID: \"fa5df00c-41f7-40b6-973d-c7090ae9dbca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.070236 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.070212    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:42.070236 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.070232    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:42.070381 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.070245    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v7fgt for pod openshift-network-diagnostics/network-check-target-kmt2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:42.070381 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.070300    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt podName:58593c9f-7d98-433a-a4cd-9507eb36928d nodeName:}" failed. No retries permitted until 2026-09-17 23:10:42.570284571 +0000 UTC m=+3.022116054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v7fgt" (UniqueName: "kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt") pod "network-check-target-kmt2w" (UID: "58593c9f-7d98-433a-a4cd-9507eb36928d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:42.071351 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.071329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vt6\" (UniqueName: \"kubernetes.io/projected/2199a550-7b6f-455c-8731-0aedbffba902-kube-api-access-n2vt6\") pod \"iptables-alerter-9h7gv\" (UID: \"2199a550-7b6f-455c-8731-0aedbffba902\") " pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:42.071530 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.071514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gz4\" (UniqueName: \"kubernetes.io/projected/0be814cf-7b90-4d33-9f35-f73fdf631402-kube-api-access-r4gz4\") pod \"node-ca-nvz6b\" (UID: \"0be814cf-7b90-4d33-9f35-f73fdf631402\") " pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:42.072816 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.072800    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2zwk\" (UniqueName: \"kubernetes.io/projected/f4e50a1d-15b6-42be-bcd4-641f72602ad1-kube-api-access-b2zwk\") pod \"multus-additional-cni-plugins-dvw78\" (UID: \"f4e50a1d-15b6-42be-bcd4-641f72602ad1\") " pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.160818 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f99df5d-824d-4474-a81b-7cda0839ae43-cni-binary-copy\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.160928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-socket-dir-parent\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.160928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-cni-multus\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.160928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-system-cni-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.160928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsnvg\" (UniqueName: \"kubernetes.io/projected/bdb8ea08-57db-405c-ac91-9de219efe014-kube-api-access-wsnvg\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:42.160928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-socket-dir-parent\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-cni-multus\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-cni-bin\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-system-cni-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.160986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-kubelet\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-etc-kubernetes\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-netns\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-cni-bin\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-cnibin\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-etc-kubernetes\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-var-lib-kubelet\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-netns\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-cnibin\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-daemon-config\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-cni-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-k8s-cni-cncf-io\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-cni-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f99df5d-824d-4474-a81b-7cda0839ae43-cni-binary-copy\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxv4\" (UniqueName: \"kubernetes.io/projected/2f99df5d-824d-4474-a81b-7cda0839ae43-kube-api-access-ksxv4\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.161341    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-multus-certs\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.161411    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:42.661396168 +0000 UTC m=+3.113227645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-multus-certs\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-host-run-k8s-cni-cncf-io\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-hostroot\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-hostroot\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-conf-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.161599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.161528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-conf-dir\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.162314 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.162263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f99df5d-824d-4474-a81b-7cda0839ae43-multus-daemon-config\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.171417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.171399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxv4\" (UniqueName: \"kubernetes.io/projected/2f99df5d-824d-4474-a81b-7cda0839ae43-kube-api-access-ksxv4\") pod \"multus-7zhxm\" (UID: \"2f99df5d-824d-4474-a81b-7cda0839ae43\") " pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.171522 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.171504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsnvg\" (UniqueName: \"kubernetes.io/projected/bdb8ea08-57db-405c-ac91-9de219efe014-kube-api-access-wsnvg\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:42.237309 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.237290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gq5q5"
Sep 17 23:10:42.246410 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.246394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5"
Sep 17 23:10:42.253653 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.253638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9h7gv"
Sep 17 23:10:42.261132 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.261114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:10:42.267408 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.267395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tbgm7"
Sep 17 23:10:42.274745 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.274730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:10:42.280641 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.280627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dvw78"
Sep 17 23:10:42.286283 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.286267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nvz6b"
Sep 17 23:10:42.290633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.290618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7zhxm"
Sep 17 23:10:42.308043 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.308025    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:42.564520 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.564450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:42.564678 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.564588    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:42.564678 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.564647    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret podName:50534f9a-9cd4-4dfb-9e76-02e8f2247ed7 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.564631298 +0000 UTC m=+4.016462777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret") pod "global-pull-secret-syncer-jkmmb" (UID: "50534f9a-9cd4-4dfb-9e76-02e8f2247ed7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:42.665675 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.665643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:42.665823 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:42.665684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:42.665823 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.665791    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:42.665823 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.665805    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:42.665823 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.665821    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:42.665987 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.665847    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v7fgt for pod openshift-network-diagnostics/network-check-target-kmt2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:42.665987 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.665856    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.665835836 +0000 UTC m=+4.117667322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:42.665987 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:42.665884    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt podName:58593c9f-7d98-433a-a4cd-9507eb36928d nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.665873342 +0000 UTC m=+4.117704822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7fgt" (UniqueName: "kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt") pod "network-check-target-kmt2w" (UID: "58593c9f-7d98-433a-a4cd-9507eb36928d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:42.949464 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:42.949439    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938d9893_bdce_4bd2_8898_0ab74c7fc325.slice/crio-e97b148dabe267909689455ef1139b0ed39f3d23193415f868719566f71cdacd WatchSource:0}: Error finding container e97b148dabe267909689455ef1139b0ed39f3d23193415f868719566f71cdacd: Status 404 returned error can't find the container with id e97b148dabe267909689455ef1139b0ed39f3d23193415f868719566f71cdacd
Sep 17 23:10:42.951255 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:42.951228    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18d7e89_9ce2_4526_a68e_00f4ef8e82fe.slice/crio-4d3e7a7c06d1d87a02e320945e4e68363ae8817083364a9381998ab74067aa77 WatchSource:0}: Error finding container 4d3e7a7c06d1d87a02e320945e4e68363ae8817083364a9381998ab74067aa77: Status 404 returned error can't find the container with id 4d3e7a7c06d1d87a02e320945e4e68363ae8817083364a9381998ab74067aa77
Sep 17 23:10:42.952801 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:42.952773    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be814cf_7b90_4d33_9f35_f73fdf631402.slice/crio-d7129603fbad8f8ec782c4fc7c7671c788df3a98082d6ff34b1f81ff56958b2e WatchSource:0}: Error finding container d7129603fbad8f8ec782c4fc7c7671c788df3a98082d6ff34b1f81ff56958b2e: Status 404 returned error can't find the container with id d7129603fbad8f8ec782c4fc7c7671c788df3a98082d6ff34b1f81ff56958b2e
Sep 17 23:10:42.955620 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:42.955561    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab75934_578e_42db_963e_c4eadf56b841.slice/crio-d77afb3a1199816227e252051da0114e2d094ffec19125df0c6edd65ba048eea WatchSource:0}: Error finding container d77afb3a1199816227e252051da0114e2d094ffec19125df0c6edd65ba048eea: Status 404 returned error can't find the container with id d77afb3a1199816227e252051da0114e2d094ffec19125df0c6edd65ba048eea
Sep 17 23:10:42.956534 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:42.956516    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3162a2e0_39fc_43da_97f6_18fd0932c183.slice/crio-02cc3d2f7fd2f91c10d2069137df10906926ac8ee8ba8c1289ebc1f55b144993 WatchSource:0}: Error finding container 02cc3d2f7fd2f91c10d2069137df10906926ac8ee8ba8c1289ebc1f55b144993: Status 404 returned error can't find the container with id 02cc3d2f7fd2f91c10d2069137df10906926ac8ee8ba8c1289ebc1f55b144993
Sep 17 23:10:42.958469 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:10:42.958445    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2199a550_7b6f_455c_8731_0aedbffba902.slice/crio-6bb89e58c898c0e9776f5da1a8e6084e9ac91f8e34bf7977e579c50c0ab6b15e WatchSource:0}: Error finding container 6bb89e58c898c0e9776f5da1a8e6084e9ac91f8e34bf7977e579c50c0ab6b15e: Status 404 returned error can't find the container with id 6bb89e58c898c0e9776f5da1a8e6084e9ac91f8e34bf7977e579c50c0ab6b15e
Sep 17 23:10:43.018427 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.018153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerStarted","Data":"297698dc09d20220bb8ef939789d37bd3ac341638b11cffa0f379d3915cb9470"}
Sep 17 23:10:43.020409 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.020386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5" event={"ID":"fa5df00c-41f7-40b6-973d-c7090ae9dbca","Type":"ContainerStarted","Data":"73fbfde14be562643c0c125ec5b5cae062d49c651f02a6112c07e03ad15cfe6b"}
Sep 17 23:10:43.021551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.021528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9h7gv" event={"ID":"2199a550-7b6f-455c-8731-0aedbffba902","Type":"ContainerStarted","Data":"6bb89e58c898c0e9776f5da1a8e6084e9ac91f8e34bf7977e579c50c0ab6b15e"}
Sep 17 23:10:43.022679 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.022649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gq5q5" event={"ID":"938d9893-bdce-4bd2-8898-0ab74c7fc325","Type":"ContainerStarted","Data":"e97b148dabe267909689455ef1139b0ed39f3d23193415f868719566f71cdacd"}
Sep 17 23:10:43.023792 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.023774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhsgn" event={"ID":"3162a2e0-39fc-43da-97f6-18fd0932c183","Type":"ContainerStarted","Data":"02cc3d2f7fd2f91c10d2069137df10906926ac8ee8ba8c1289ebc1f55b144993"}
Sep 17 23:10:43.024312 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.024273    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:41 +0000 UTC" deadline="2028-02-13 04:18:31.290577908 +0000 UTC"
Sep 17 23:10:43.024312 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.024311    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12317h7m48.266270411s"
Sep 17 23:10:43.024882 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.024854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tbgm7" event={"ID":"fab75934-578e-42db-963e-c4eadf56b841","Type":"ContainerStarted","Data":"d77afb3a1199816227e252051da0114e2d094ffec19125df0c6edd65ba048eea"}
Sep 17 23:10:43.026022 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.026000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nvz6b" event={"ID":"0be814cf-7b90-4d33-9f35-f73fdf631402","Type":"ContainerStarted","Data":"d7129603fbad8f8ec782c4fc7c7671c788df3a98082d6ff34b1f81ff56958b2e"}
Sep 17 23:10:43.027164 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.027143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"4d3e7a7c06d1d87a02e320945e4e68363ae8817083364a9381998ab74067aa77"}
Sep 17 23:10:43.028271 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.028251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7zhxm" event={"ID":"2f99df5d-824d-4474-a81b-7cda0839ae43","Type":"ContainerStarted","Data":"29bc573dacb5b15ff44768d5f05cdf2215d230b7ce7fe614de9d87cac315193d"}
Sep 17 23:10:43.572646 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.572342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:43.572646 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.572515    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:43.572646 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.572603    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret podName:50534f9a-9cd4-4dfb-9e76-02e8f2247ed7 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:45.572568018 +0000 UTC m=+6.024399491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret") pod "global-pull-secret-syncer-jkmmb" (UID: "50534f9a-9cd4-4dfb-9e76-02e8f2247ed7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.673115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:43.673186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.673319    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.673377    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:45.673358439 +0000 UTC m=+6.125189927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.673779    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.673800    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.673811    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v7fgt for pod openshift-network-diagnostics/network-check-target-kmt2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:43.673887 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:43.673854    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt podName:58593c9f-7d98-433a-a4cd-9507eb36928d nodeName:}" failed. No retries permitted until 2026-09-17 23:10:45.673840618 +0000 UTC m=+6.125672087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7fgt" (UniqueName: "kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt") pod "network-check-target-kmt2w" (UID: "58593c9f-7d98-433a-a4cd-9507eb36928d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:44.010938 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.010871    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:44.011593 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:44.011002    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:44.011593 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.011382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:44.011593 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:44.011466    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:44.011593 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.011520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:44.011809 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:44.011622    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:44.024626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.024594    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:41 +0000 UTC" deadline="2028-05-17 10:12:43.327629715 +0000 UTC"
Sep 17 23:10:44.024626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.024627    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14579h1m59.30300734s"
Sep 17 23:10:44.057394 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.057364    2566 generic.go:356] "Generic (PLEG): container finished" podID="52fd5495c101430ae182aa561a48cae8" containerID="549b6f27c27b8bf8b0e8912975ea90b3897e3ce4debc20b27f4286f2393ca527" exitCode=0
Sep 17 23:10:44.057497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.057452    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal" event={"ID":"52fd5495c101430ae182aa561a48cae8","Type":"ContainerDied","Data":"549b6f27c27b8bf8b0e8912975ea90b3897e3ce4debc20b27f4286f2393ca527"}
Sep 17 23:10:44.064307 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:44.064282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal" event={"ID":"b015008b02def3f3ba52667fd2a6cf91","Type":"ContainerStarted","Data":"1d0bf6c8615c84208c5454b6082b5c1248f45aa19b45214ecf864cc0c13dbf06"}
Sep 17 23:10:45.082162 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:45.082123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal" event={"ID":"52fd5495c101430ae182aa561a48cae8","Type":"ContainerStarted","Data":"9e59463bc7f4b0b80623e1134cab98c8f07e071ea10315da89bddd7c858ec228"}
Sep 17 23:10:45.097693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:45.097643    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-66.ec2.internal" podStartSLOduration=4.097621591 podStartE2EDuration="4.097621591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:44.085686173 +0000 UTC m=+4.537517662" watchObservedRunningTime="2026-09-17 23:10:45.097621591 +0000 UTC m=+5.549453072"
Sep 17 23:10:45.098026 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:45.097991    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-66.ec2.internal" podStartSLOduration=4.097981252 podStartE2EDuration="4.097981252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:45.096444427 +0000 UTC m=+5.548275916" watchObservedRunningTime="2026-09-17 23:10:45.097981252 +0000 UTC m=+5.549812743"
Sep 17 23:10:45.591118 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:45.591047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:45.591402 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.591199    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:45.591402 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.591275    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret podName:50534f9a-9cd4-4dfb-9e76-02e8f2247ed7 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.591256305 +0000 UTC m=+10.043087774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret") pod "global-pull-secret-syncer-jkmmb" (UID: "50534f9a-9cd4-4dfb-9e76-02e8f2247ed7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:45.691902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:45.691868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:45.692045 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:45.691929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:45.692102 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.692066    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:45.692165 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.692124    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.692106742 +0000 UTC m=+10.143938225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:45.692531 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.692511    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:45.692632 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.692536    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:45.692632 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.692548    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v7fgt for pod openshift-network-diagnostics/network-check-target-kmt2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:45.692632 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:45.692609    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt podName:58593c9f-7d98-433a-a4cd-9507eb36928d nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.692594283 +0000 UTC m=+10.144425772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7fgt" (UniqueName: "kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt") pod "network-check-target-kmt2w" (UID: "58593c9f-7d98-433a-a4cd-9507eb36928d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:46.008648 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:46.008540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:46.008648 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:46.008617    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:46.008847 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:46.008543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:46.008847 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:46.008747    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:46.008847 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:46.008807    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:46.009142 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:46.009094    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:48.009599 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:48.009445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:48.009599 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:48.009590    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:48.010072 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:48.009690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:48.010072 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:48.009809    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:48.010072 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:48.009869    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:48.010072 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:48.009937    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:49.623605 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:49.623491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:49.624064 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.623654    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.624064 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.623721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret podName:50534f9a-9cd4-4dfb-9e76-02e8f2247ed7 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:57.623701888 +0000 UTC m=+18.075533353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret") pod "global-pull-secret-syncer-jkmmb" (UID: "50534f9a-9cd4-4dfb-9e76-02e8f2247ed7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.723929 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:49.723870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:49.723929 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:49.723919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:49.724160 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.724013    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.724160 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.724023    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:49.724160 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.724042    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:49.724160 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.724056    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v7fgt for pod openshift-network-diagnostics/network-check-target-kmt2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.724160 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.724075    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:57.724057146 +0000 UTC m=+18.175888626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.724160 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:49.724106    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt podName:58593c9f-7d98-433a-a4cd-9507eb36928d nodeName:}" failed. No retries permitted until 2026-09-17 23:10:57.724089608 +0000 UTC m=+18.175921099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7fgt" (UniqueName: "kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt") pod "network-check-target-kmt2w" (UID: "58593c9f-7d98-433a-a4cd-9507eb36928d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:50.011073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:50.011003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:50.011477 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:50.011446    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:50.012160 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:50.012142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:50.012438 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:50.012417    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:50.012776 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:50.012762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:50.013055 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:50.013035    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:52.008202 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:52.008168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:52.008782 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:52.008168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:52.008782 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:52.008329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:52.008782 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:52.008291    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:52.008782 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:52.008418    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:52.008782 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:52.008469    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:54.008379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:54.008345    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:54.008379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:54.008369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:54.008897 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:54.008473    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:54.008897 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:54.008492    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:54.008897 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:54.008604    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:54.008897 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:54.008738    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:56.008278 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:56.008246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:56.008278 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:56.008270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:56.008783 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:56.008369    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:56.008783 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:56.008439    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:10:56.008783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:56.008497    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:56.008783 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:56.008568    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:57.684934 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:57.684901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:57.685450 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.685080    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:57.685450 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.685155    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret podName:50534f9a-9cd4-4dfb-9e76-02e8f2247ed7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.685135241 +0000 UTC m=+34.136966711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret") pod "global-pull-secret-syncer-jkmmb" (UID: "50534f9a-9cd4-4dfb-9e76-02e8f2247ed7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:57.786176 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:57.786143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:57.786330 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:57.786196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:57.786330 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.786301    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:57.786330 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.786312    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:57.786330 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.786333    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:57.786519 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.786344    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v7fgt for pod openshift-network-diagnostics/network-check-target-kmt2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:57.786519 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.786360    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.786341848 +0000 UTC m=+34.238173315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:57.786519 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:57.786394    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt podName:58593c9f-7d98-433a-a4cd-9507eb36928d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.786377817 +0000 UTC m=+34.238209312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7fgt" (UniqueName: "kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt") pod "network-check-target-kmt2w" (UID: "58593c9f-7d98-433a-a4cd-9507eb36928d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:58.008622 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:58.008535    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:10:58.008777 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:58.008538    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:10:58.008777 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:58.008663    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:10:58.008777 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:58.008711    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:10:58.008777 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:10:58.008549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:10:58.008967 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:10:58.008815    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:00.008966 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:00.008934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:00.009387 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:00.009040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:00.009387 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:00.009053    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:00.009387 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:00.009158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:00.009387 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:00.009182    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:00.009387 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:00.009225    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:01.112987 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:01.112957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhsgn" event={"ID":"3162a2e0-39fc-43da-97f6-18fd0932c183","Type":"ContainerStarted","Data":"48b6f421435985a62ead8cc2b5a0fd1f250be3f0e89d2b60193db9e3f2b70a52"}
Sep 17 23:11:01.127319 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:01.127148    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhsgn" podStartSLOduration=7.157435884 podStartE2EDuration="21.127128588s" totalImagesPullingTime="13.969692704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.126253689 +0000 UTC m=+21.578085199" watchObservedRunningTime="2026-09-17 23:11:01.127128588 +0000 UTC m=+21.578960077"
Sep 17 23:11:02.008768 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.008606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:02.008931 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.008606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:02.008931 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:02.008841    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:02.009045 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:02.008948    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:02.009045 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.008659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:02.009126 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:02.009054    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:02.120947 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.120884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tbgm7" event={"ID":"fab75934-578e-42db-963e-c4eadf56b841","Type":"ContainerStarted","Data":"d53f85ceaae345dae9f846a184bdb3ba58bf022ca4bfc4575ed51eac10034647"}
Sep 17 23:11:02.122376 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.122355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nvz6b" event={"ID":"0be814cf-7b90-4d33-9f35-f73fdf631402","Type":"ContainerStarted","Data":"a7c2967150b28bc30601114814d928027e854b684f835bc4c3ffd046cd954963"}
Sep 17 23:11:02.124925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.124908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:11:02.125260 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.125243    2566 generic.go:356] "Generic (PLEG): container finished" podID="e18d7e89-9ce2-4526-a68e-00f4ef8e82fe" containerID="b92ec98de4cdfded11d94b6cc173faf78ecd4be1322c37fe3bb34bf8c1879822" exitCode=1
Sep 17 23:11:02.125323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.125307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"da78f942295cf6018b906acd7b7968d272d1eeb8aee8770ff9975375503a2ccb"}
Sep 17 23:11:02.125381 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.125369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"7eab3f2cd433ce326ff7cd385cbf573ed94570b9221c046a14813c9d8e53418c"}
Sep 17 23:11:02.125419 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.125408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"65dc40695e3273d707461e047d5b4cee6af3318cc9a3accf73482f981d415d0f"}
Sep 17 23:11:02.125451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.125441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"8a931740785413efc23325b6f96a5944b8f5056af1ee3ad99dc14a379ee016d0"}
Sep 17 23:11:02.125480 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.125468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerDied","Data":"b92ec98de4cdfded11d94b6cc173faf78ecd4be1322c37fe3bb34bf8c1879822"}
Sep 17 23:11:02.125528 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.125506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"e8f1916f1c0e55d4e2ebd79c8d35f8cac71533b7ba78a3acd0a08b9753e6d208"}
Sep 17 23:11:02.126674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.126651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7zhxm" event={"ID":"2f99df5d-824d-4474-a81b-7cda0839ae43","Type":"ContainerStarted","Data":"ec31108dd04157a02d4ef0fdf26dda2bc79ba9370f6ca4fc7cf913fbd41959a9"}
Sep 17 23:11:02.128075 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.128056    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4e50a1d-15b6-42be-bcd4-641f72602ad1" containerID="1de72643a46267b0b22ee9cf38f1940d8ff20683fbfbc946559329bbfc584e17" exitCode=0
Sep 17 23:11:02.128175 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.128118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerDied","Data":"1de72643a46267b0b22ee9cf38f1940d8ff20683fbfbc946559329bbfc584e17"}
Sep 17 23:11:02.129473 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.129454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5" event={"ID":"fa5df00c-41f7-40b6-973d-c7090ae9dbca","Type":"ContainerStarted","Data":"1a7c1b6b2586aee1813d90bd71340ad3f3dc85728be732846ec43210d26fab01"}
Sep 17 23:11:02.130726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.130706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gq5q5" event={"ID":"938d9893-bdce-4bd2-8898-0ab74c7fc325","Type":"ContainerStarted","Data":"1238ae25d15f510d2408fe9a91e4c0a64065ac31b18f21dc238e9259f210d05b"}
Sep 17 23:11:02.150348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.150299    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tbgm7" podStartSLOduration=4.202041227 podStartE2EDuration="22.150284393s" totalImagesPullingTime="17.948243166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.136999059 +0000 UTC m=+22.588830547" watchObservedRunningTime="2026-09-17 23:11:02.150284393 +0000 UTC m=+22.602115882"
Sep 17 23:11:02.150451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.150426    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gq5q5" podStartSLOduration=5.196062992 podStartE2EDuration="23.150418276s" totalImagesPullingTime="17.954355284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.150370153 +0000 UTC m=+22.602201642" watchObservedRunningTime="2026-09-17 23:11:02.150418276 +0000 UTC m=+22.602249766"
Sep 17 23:11:02.165171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.165138    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nvz6b" podStartSLOduration=4.213493047 podStartE2EDuration="22.165129567s" totalImagesPullingTime="17.95163652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.164855262 +0000 UTC m=+22.616686752" watchObservedRunningTime="2026-09-17 23:11:02.165129567 +0000 UTC m=+22.616961055"
Sep 17 23:11:02.206469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.206436    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7zhxm" podStartSLOduration=3.76528063 podStartE2EDuration="22.20642692s" totalImagesPullingTime="18.44114629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.206258889 +0000 UTC m=+22.658090376" watchObservedRunningTime="2026-09-17 23:11:02.20642692 +0000 UTC m=+22.658258408"
Sep 17 23:11:02.385428 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:02.385407    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:11:03.015967 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:03.015852    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:11:02.385424385Z","UUID":"177b86bf-7157-4698-8262-b5965d85f7d8","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:11:03.018199 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:03.018176    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:11:03.018336 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:03.018213    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:11:03.135341 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:03.135300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5" event={"ID":"fa5df00c-41f7-40b6-973d-c7090ae9dbca","Type":"ContainerStarted","Data":"e247a903c9592574101983ad83c8a356bb9cf35ddf47e8d20a90cc8f9a041ef5"}
Sep 17 23:11:03.137342 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:03.137304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9h7gv" event={"ID":"2199a550-7b6f-455c-8731-0aedbffba902","Type":"ContainerStarted","Data":"4ea5bc87124766e068bcd4d2562e99610cf6e8f1c3c11a36ff5f0da13c0e285d"}
Sep 17 23:11:04.009067 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.008990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:04.009325 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.008996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:04.009325 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:04.009097    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:04.009325 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.009119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:04.009325 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:04.009201    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:04.009325 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:04.009304    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:04.141564 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.141531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5" event={"ID":"fa5df00c-41f7-40b6-973d-c7090ae9dbca","Type":"ContainerStarted","Data":"04ffab4c4c83e24670e982a24f84d76ef6c6100892d01c23947920bb61c83cc9"}
Sep 17 23:11:04.145033 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.145012    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:11:04.145493 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.145472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"82668247f0d87d2710319a8d4888bc8937e678ed27c8e19b7e1b034846dc6fb3"}
Sep 17 23:11:04.158940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.158898    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdd5" podStartSLOduration=3.893338264 podStartE2EDuration="24.158881654s" totalImagesPullingTime="20.26554339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:04.158717568 +0000 UTC m=+24.610549057" watchObservedRunningTime="2026-09-17 23:11:04.158881654 +0000 UTC m=+24.610713144"
Sep 17 23:11:04.159397 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:04.159365    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9h7gv" podStartSLOduration=7.213856953 podStartE2EDuration="25.159357983s" totalImagesPullingTime="17.94550103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:03.153324255 +0000 UTC m=+23.605155744" watchObservedRunningTime="2026-09-17 23:11:04.159357983 +0000 UTC m=+24.611189471"
Sep 17 23:11:05.742985 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:05.742752    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:11:05.743454 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:05.743338    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:11:06.008911 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:06.008846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:06.009050 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:06.008860    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:06.009050 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:06.008965    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:06.009178 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:06.009043    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:06.009178 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:06.009103    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:06.009257 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:06.009184    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:06.149620 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:06.149593    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:11:06.150131 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:06.150113    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhsgn"
Sep 17 23:11:07.162129 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.161937    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4e50a1d-15b6-42be-bcd4-641f72602ad1" containerID="7171d33e22182172ebb21e62c1bc5ce44a21a02536e0986575283f031dea31d6" exitCode=0
Sep 17 23:11:07.162539 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.162021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerDied","Data":"7171d33e22182172ebb21e62c1bc5ce44a21a02536e0986575283f031dea31d6"}
Sep 17 23:11:07.166604 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.166565    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:11:07.167060 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.167033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"f5b65eaaf1f71d0e46c976847c0c1ad6eac6bf77c9f620388be35a89ce23c72c"}
Sep 17 23:11:07.167551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.167447    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:11:07.167551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.167522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:11:07.167551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.167533    2566 scope.go:117] "RemoveContainer" containerID="b92ec98de4cdfded11d94b6cc173faf78ecd4be1322c37fe3bb34bf8c1879822"
Sep 17 23:11:07.167771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.167598    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:11:07.184714 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.184690    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:11:07.184902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:07.184875    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:11:08.008385 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.008336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:08.008513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.008344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:08.008513 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:08.008429    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:08.008513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.008437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:08.008637 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:08.008540    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:08.008681 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:08.008661    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:08.161964 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.161942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkmmb"]
Sep 17 23:11:08.162879 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.162857    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kmt2w"]
Sep 17 23:11:08.167685 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.167668    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zd5lf"]
Sep 17 23:11:08.171735 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.171715    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4e50a1d-15b6-42be-bcd4-641f72602ad1" containerID="89f93ff48d4354ba9e02162c31b1bab1eb1f5936aa2108355d8682e4f3ce205d" exitCode=0
Sep 17 23:11:08.171822 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.171789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerDied","Data":"89f93ff48d4354ba9e02162c31b1bab1eb1f5936aa2108355d8682e4f3ce205d"}
Sep 17 23:11:08.176374 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.176358    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:11:08.176771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.176754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:08.176771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.176744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" event={"ID":"e18d7e89-9ce2-4526-a68e-00f4ef8e82fe","Type":"ContainerStarted","Data":"ba951030d26c61c66455e30859c24814306c95c732a7ce88dcf6eb9cbf5d55e1"}
Sep 17 23:11:08.176909 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:08.176858    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:08.177240 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.177223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:08.177321 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:08.177298    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:08.177321 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.177305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:08.177428 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:08.177363    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:08.218074 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:08.218036    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw" podStartSLOduration=10.225405839 podStartE2EDuration="28.218024783s" totalImagesPullingTime="17.992618944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:08.21625488 +0000 UTC m=+28.668086367" watchObservedRunningTime="2026-09-17 23:11:08.218024783 +0000 UTC m=+28.669856271"
Sep 17 23:11:09.181661 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:09.181406    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4e50a1d-15b6-42be-bcd4-641f72602ad1" containerID="901c05eb491b4cf88e7a1a4b4f7bb99b5f5ca01ba9b7c6fdb5226df96d21a47c" exitCode=0
Sep 17 23:11:09.182087 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:09.181493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerDied","Data":"901c05eb491b4cf88e7a1a4b4f7bb99b5f5ca01ba9b7c6fdb5226df96d21a47c"}
Sep 17 23:11:10.009473 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:10.009431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:10.009619 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:10.009550    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:10.009619 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:10.009566    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:10.009751 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:10.009672    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:10.009751 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:10.009739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:10.009843 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:10.009827    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:12.008694 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:12.008662    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:12.009135 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:12.008669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:12.009135 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:12.008796    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:12.009135 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:12.008870    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:12.009135 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:12.008942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:12.009135 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:12.009041    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:13.699323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:13.699257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:13.699804 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.699392    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:13.699804 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.699461    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret podName:50534f9a-9cd4-4dfb-9e76-02e8f2247ed7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.699443877 +0000 UTC m=+66.151275357 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret") pod "global-pull-secret-syncer-jkmmb" (UID: "50534f9a-9cd4-4dfb-9e76-02e8f2247ed7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:13.799530 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:13.799491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:13.799688 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:13.799544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:13.799688 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.799663    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:13.799780 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.799696    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:13.799780 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.799751    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v7fgt for pod openshift-network-diagnostics/network-check-target-kmt2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:13.799780 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.799665    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:13.799869 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.799812    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.79979631 +0000 UTC m=+66.251627775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:13.799869 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:13.799842    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt podName:58593c9f-7d98-433a-a4cd-9507eb36928d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.799818811 +0000 UTC m=+66.251650297 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7fgt" (UniqueName: "kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt") pod "network-check-target-kmt2w" (UID: "58593c9f-7d98-433a-a4cd-9507eb36928d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:14.008367 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.008311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:14.008367 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.008330    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:14.008367 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.008332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:14.008677 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:14.008402    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkmmb" podUID="50534f9a-9cd4-4dfb-9e76-02e8f2247ed7"
Sep 17 23:11:14.008677 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:14.008501    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:11:14.008677 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:14.008592    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmt2w" podUID="58593c9f-7d98-433a-a4cd-9507eb36928d"
Sep 17 23:11:14.196410 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.196380    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4e50a1d-15b6-42be-bcd4-641f72602ad1" containerID="95d3bfb0aa46a9a4224b3682758216c9345cfc49fbeecd62482fa1e9ef7b4a44" exitCode=0
Sep 17 23:11:14.196609 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.196442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerDied","Data":"95d3bfb0aa46a9a4224b3682758216c9345cfc49fbeecd62482fa1e9ef7b4a44"}
Sep 17 23:11:14.345540 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.345520    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-66.ec2.internal" event="NodeReady"
Sep 17 23:11:14.345670 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.345658    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:11:14.415604 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.415567    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zzxvm"]
Sep 17 23:11:14.426354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.426336    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjz88"]
Sep 17 23:11:14.426468 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.426450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.429741 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.429724    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvr9f\""
Sep 17 23:11:14.429946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.429927    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:11:14.431674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.431658    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:11:14.438496 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.438480    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:14.443213 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.443193    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:11:14.443320 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.443235    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:11:14.443320 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.443264    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:11:14.443445 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.443424    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fsw47\""
Sep 17 23:11:14.447919 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.447900    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzxvm"]
Sep 17 23:11:14.448469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.448449    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjz88"]
Sep 17 23:11:14.604381 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.604329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-tmp-dir\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.604381 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.604365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e92de406-eab7-42ca-a693-611ad02f1377-cert\") pod \"ingress-canary-hjz88\" (UID: \"e92de406-eab7-42ca-a693-611ad02f1377\") " pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:14.604497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.604383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-config-volume\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.604497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.604414    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.604497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.604457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbprc\" (UniqueName: \"kubernetes.io/projected/e92de406-eab7-42ca-a693-611ad02f1377-kube-api-access-tbprc\") pod \"ingress-canary-hjz88\" (UID: \"e92de406-eab7-42ca-a693-611ad02f1377\") " pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:14.604497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.604478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzb5h\" (UniqueName: \"kubernetes.io/projected/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-kube-api-access-gzb5h\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.705607 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.705565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-config-volume\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.705903 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.705616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.705903 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.705647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbprc\" (UniqueName: \"kubernetes.io/projected/e92de406-eab7-42ca-a693-611ad02f1377-kube-api-access-tbprc\") pod \"ingress-canary-hjz88\" (UID: \"e92de406-eab7-42ca-a693-611ad02f1377\") " pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:14.705903 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.705690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzb5h\" (UniqueName: \"kubernetes.io/projected/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-kube-api-access-gzb5h\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.705903 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.705737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-tmp-dir\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.705903 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.705768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e92de406-eab7-42ca-a693-611ad02f1377-cert\") pod \"ingress-canary-hjz88\" (UID: \"e92de406-eab7-42ca-a693-611ad02f1377\") " pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:14.705903 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:14.705783    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:14.705903 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:14.705850    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:15.20583249 +0000 UTC m=+35.657663991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:14.706236 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.706077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-tmp-dir\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.706236 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.706134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-config-volume\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.710373 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.710352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e92de406-eab7-42ca-a693-611ad02f1377-cert\") pod \"ingress-canary-hjz88\" (UID: \"e92de406-eab7-42ca-a693-611ad02f1377\") " pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:14.717517 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.717489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbprc\" (UniqueName: \"kubernetes.io/projected/e92de406-eab7-42ca-a693-611ad02f1377-kube-api-access-tbprc\") pod \"ingress-canary-hjz88\" (UID: \"e92de406-eab7-42ca-a693-611ad02f1377\") " pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:14.717797 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.717779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzb5h\" (UniqueName: \"kubernetes.io/projected/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-kube-api-access-gzb5h\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:14.749939 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:14.749917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjz88"
Sep 17 23:11:15.070844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:15.070775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjz88"]
Sep 17 23:11:15.075779 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:11:15.075753    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92de406_eab7_42ca_a693_611ad02f1377.slice/crio-eb3c4bcae925d9212bad4554d0bbfe20b75a6ecb0276e2e97ad95e6c5274e7ad WatchSource:0}: Error finding container eb3c4bcae925d9212bad4554d0bbfe20b75a6ecb0276e2e97ad95e6c5274e7ad: Status 404 returned error can't find the container with id eb3c4bcae925d9212bad4554d0bbfe20b75a6ecb0276e2e97ad95e6c5274e7ad
Sep 17 23:11:15.202058 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:15.202031    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4e50a1d-15b6-42be-bcd4-641f72602ad1" containerID="49a128593b02b778705e266a699013a51977844b412eefa1acfc50efdbd6b6ec" exitCode=0
Sep 17 23:11:15.202158 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:15.202100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerDied","Data":"49a128593b02b778705e266a699013a51977844b412eefa1acfc50efdbd6b6ec"}
Sep 17 23:11:15.203298 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:15.203275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjz88" event={"ID":"e92de406-eab7-42ca-a693-611ad02f1377","Type":"ContainerStarted","Data":"eb3c4bcae925d9212bad4554d0bbfe20b75a6ecb0276e2e97ad95e6c5274e7ad"}
Sep 17 23:11:15.210982 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:15.210959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:15.211121 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:15.211104    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:15.211168 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:15.211159    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:16.211143953 +0000 UTC m=+36.662975418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:16.008550 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.008517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:16.009282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.008557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:16.009282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.008517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:16.012096 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.011911    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:16.013308 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.013285    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:16.013448 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.013323    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:16.013448 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.013412    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qw6rm\""
Sep 17 23:11:16.013448 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.013327    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:16.013973 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.013650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mx9lw\""
Sep 17 23:11:16.210558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.210521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvw78" event={"ID":"f4e50a1d-15b6-42be-bcd4-641f72602ad1","Type":"ContainerStarted","Data":"ce28d4766cbf1cd8e913c6110a17e8ade151a15674668822a7e79b4890abdb1b"}
Sep 17 23:11:16.216200 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.216179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:16.216323 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:16.216287    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:16.216386 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:16.216378    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:18.216358156 +0000 UTC m=+38.668189635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:16.236405 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:16.236362    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dvw78" podStartSLOduration=8.973561253 podStartE2EDuration="36.236346544s" totalImagesPullingTime="27.262785291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:16.235149147 +0000 UTC m=+36.686980635" watchObservedRunningTime="2026-09-17 23:11:16.236346544 +0000 UTC m=+36.688178035"
Sep 17 23:11:18.216547 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:18.216471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjz88" event={"ID":"e92de406-eab7-42ca-a693-611ad02f1377","Type":"ContainerStarted","Data":"b2ffa2584ad5e8687189deb215551c8ed01ae105091fb369e545d6727cc5ac81"}
Sep 17 23:11:18.231355 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:18.231330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:18.231455 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:18.231436    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:18.231489 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:18.231482    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:22.231469293 +0000 UTC m=+42.683300759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:18.232514 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:18.232480    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjz88" podStartSLOduration=1.818169754 podStartE2EDuration="4.232467079s" totalImagesPullingTime="2.414297325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:18.230935949 +0000 UTC m=+38.682767437" watchObservedRunningTime="2026-09-17 23:11:18.232467079 +0000 UTC m=+38.684298566"
Sep 17 23:11:22.261781 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:22.261545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:22.262222 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:22.261698    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:22.262222 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:22.261867    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:30.261850609 +0000 UTC m=+50.713682118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:30.311257 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:30.311209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:30.311760 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:30.311350    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:30.311760 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:30.311411    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:11:46.311395624 +0000 UTC m=+66.763227089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:39.193947 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:39.193908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rgw"
Sep 17 23:11:45.715770 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.715723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:45.718390 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.718363    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:45.730002 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.729975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50534f9a-9cd4-4dfb-9e76-02e8f2247ed7-original-pull-secret\") pod \"global-pull-secret-syncer-jkmmb\" (UID: \"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7\") " pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:45.816367 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.816338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:11:45.816465 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.816395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:45.819306 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.819285    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:45.819359 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.819304    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:45.826647 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:45.826627    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:11:45.826721 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:45.826707    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:49.826684656 +0000 UTC m=+130.278516127 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : secret "metrics-daemon-secret" not found
Sep 17 23:11:45.828764 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.828743    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:45.839029 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:45.839010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7fgt\" (UniqueName: \"kubernetes.io/projected/58593c9f-7d98-433a-a4cd-9507eb36928d-kube-api-access-v7fgt\") pod \"network-check-target-kmt2w\" (UID: \"58593c9f-7d98-433a-a4cd-9507eb36928d\") " pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:46.022550 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:46.022529    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qw6rm\""
Sep 17 23:11:46.027735 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:46.027721    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkmmb"
Sep 17 23:11:46.030479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:46.030458    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:46.309020 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:46.308952    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkmmb"]
Sep 17 23:11:46.312408 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:46.312384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kmt2w"]
Sep 17 23:11:46.314529 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:11:46.312729    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50534f9a_9cd4_4dfb_9e76_02e8f2247ed7.slice/crio-fcf6c65763c82e263de7825acf5fc8596161a12bcad39bcecb74559f2b892e4f WatchSource:0}: Error finding container fcf6c65763c82e263de7825acf5fc8596161a12bcad39bcecb74559f2b892e4f: Status 404 returned error can't find the container with id fcf6c65763c82e263de7825acf5fc8596161a12bcad39bcecb74559f2b892e4f
Sep 17 23:11:46.316271 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:11:46.316244    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58593c9f_7d98_433a_a4cd_9507eb36928d.slice/crio-959af416d35a29d19ad23b5fe2113babcfdbdfeb64ae898f06677e29abbd44d1 WatchSource:0}: Error finding container 959af416d35a29d19ad23b5fe2113babcfdbdfeb64ae898f06677e29abbd44d1: Status 404 returned error can't find the container with id 959af416d35a29d19ad23b5fe2113babcfdbdfeb64ae898f06677e29abbd44d1
Sep 17 23:11:46.320790 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:46.320768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:11:46.320899 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:46.320887    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:46.320939 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:11:46.320932    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:12:18.320919416 +0000 UTC m=+98.772750896 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:47.287844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:47.287804    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kmt2w" event={"ID":"58593c9f-7d98-433a-a4cd-9507eb36928d","Type":"ContainerStarted","Data":"959af416d35a29d19ad23b5fe2113babcfdbdfeb64ae898f06677e29abbd44d1"}
Sep 17 23:11:47.289338 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:47.289304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkmmb" event={"ID":"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7","Type":"ContainerStarted","Data":"fcf6c65763c82e263de7825acf5fc8596161a12bcad39bcecb74559f2b892e4f"}
Sep 17 23:11:51.302291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:51.302252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kmt2w" event={"ID":"58593c9f-7d98-433a-a4cd-9507eb36928d","Type":"ContainerStarted","Data":"6f2dd5d47e8949fd4559b4226dbaa6feca5cc7d4efcd50aef1af3a7836833364"}
Sep 17 23:11:51.302740 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:51.302376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:11:51.304045 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:51.304021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkmmb" event={"ID":"50534f9a-9cd4-4dfb-9e76-02e8f2247ed7","Type":"ContainerStarted","Data":"8088ad5ffedf8b083399ea9f2a8c040a2b9e555cd8389c9750e1eee5f83df8b3"}
Sep 17 23:11:51.318349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:51.318305    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kmt2w" podStartSLOduration=66.783732201 podStartE2EDuration="1m11.318293749s" totalImagesPullingTime="4.534561548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:51.318120282 +0000 UTC m=+71.769951770" watchObservedRunningTime="2026-09-17 23:11:51.318293749 +0000 UTC m=+71.770125236"
Sep 17 23:11:51.332977 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:11:51.332934    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jkmmb" podStartSLOduration=66.793396213 podStartE2EDuration="1m11.332918033s" totalImagesPullingTime="4.53952182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:51.332739665 +0000 UTC m=+71.784571153" watchObservedRunningTime="2026-09-17 23:11:51.332918033 +0000 UTC m=+71.784749522"
Sep 17 23:12:18.327334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:12:18.327296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:12:18.327767 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:12:18.327419    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:12:18.327767 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:12:18.327479    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:13:22.327466402 +0000 UTC m=+162.779297868 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:22.308811 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:12:22.308781    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kmt2w"
Sep 17 23:12:49.922663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:12:49.922619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:12:49.923188 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:12:49.922750    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:12:49.923188 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:12:49.922822    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs podName:bdb8ea08-57db-405c-ac91-9de219efe014 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:51.922804594 +0000 UTC m=+252.374636061 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs") pod "network-metrics-daemon-zd5lf" (UID: "bdb8ea08-57db-405c-ac91-9de219efe014") : secret "metrics-daemon-secret" not found
Sep 17 23:13:17.444053 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:13:17.444013    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zzxvm" podUID="6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d"
Sep 17 23:13:17.515938 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:17.515918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:13:19.034081 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:13:19.034036    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zd5lf" podUID="bdb8ea08-57db-405c-ac91-9de219efe014"
Sep 17 23:13:22.028102 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.028066    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"]
Sep 17 23:13:22.029962 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.029941    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.033011 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.032986    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 23:13:22.034614 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.034566    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 23:13:22.034742 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.034630    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 23:13:22.034742 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.034663    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-skx5k\""
Sep 17 23:13:22.034742 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.034597    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 23:13:22.048050 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.048027    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"]
Sep 17 23:13:22.133301 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.133270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc5e5836-d228-4823-bdba-f6cff18aa1d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw\" (UID: \"fc5e5836-d228-4823-bdba-f6cff18aa1d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.133413 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.133309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d95v\" (UniqueName: \"kubernetes.io/projected/fc5e5836-d228-4823-bdba-f6cff18aa1d3-kube-api-access-4d95v\") pod \"managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw\" (UID: \"fc5e5836-d228-4823-bdba-f6cff18aa1d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.234415 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.234391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc5e5836-d228-4823-bdba-f6cff18aa1d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw\" (UID: \"fc5e5836-d228-4823-bdba-f6cff18aa1d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.234495 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.234420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d95v\" (UniqueName: \"kubernetes.io/projected/fc5e5836-d228-4823-bdba-f6cff18aa1d3-kube-api-access-4d95v\") pod \"managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw\" (UID: \"fc5e5836-d228-4823-bdba-f6cff18aa1d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.237206 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.237181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc5e5836-d228-4823-bdba-f6cff18aa1d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw\" (UID: \"fc5e5836-d228-4823-bdba-f6cff18aa1d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.243300 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.243264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d95v\" (UniqueName: \"kubernetes.io/projected/fc5e5836-d228-4823-bdba-f6cff18aa1d3-kube-api-access-4d95v\") pod \"managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw\" (UID: \"fc5e5836-d228-4823-bdba-f6cff18aa1d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.334795 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.334737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:13:22.334887 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:13:22.334822    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:13:22.334932 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:13:22.334907    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls podName:6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d nodeName:}" failed. No retries permitted until 2026-09-17 23:15:24.334893795 +0000 UTC m=+284.786725260 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls") pod "dns-default-zzxvm" (UID: "6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d") : secret "dns-default-metrics-tls" not found
Sep 17 23:13:22.347010 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.346984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"
Sep 17 23:13:22.615640 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:22.615557    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw"]
Sep 17 23:13:22.619073 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:13:22.619047    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc5e5836_d228_4823_bdba_f6cff18aa1d3.slice/crio-db2deda2068003408e8fff7b9f3533acc8a1d0c9eaf8b0a63ba0b21a8ead1137 WatchSource:0}: Error finding container db2deda2068003408e8fff7b9f3533acc8a1d0c9eaf8b0a63ba0b21a8ead1137: Status 404 returned error can't find the container with id db2deda2068003408e8fff7b9f3533acc8a1d0c9eaf8b0a63ba0b21a8ead1137
Sep 17 23:13:23.532641 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:23.532601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw" event={"ID":"fc5e5836-d228-4823-bdba-f6cff18aa1d3","Type":"ContainerStarted","Data":"db2deda2068003408e8fff7b9f3533acc8a1d0c9eaf8b0a63ba0b21a8ead1137"}
Sep 17 23:13:25.540079 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:25.540038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw" event={"ID":"fc5e5836-d228-4823-bdba-f6cff18aa1d3","Type":"ContainerStarted","Data":"bad7c0998acb9b97186ad8d729842b9f4538427481439f6f7a889f0207da0c8b"}
Sep 17 23:13:25.557768 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:25.557720    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw" podStartSLOduration=1.026550924 podStartE2EDuration="3.55770298s" totalImagesPullingTime="2.531152056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:25.557048921 +0000 UTC m=+166.008880409" watchObservedRunningTime="2026-09-17 23:13:25.55770298 +0000 UTC m=+166.009534468"
Sep 17 23:13:32.008794 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:13:32.008751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:14:30.091168 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.091133    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-hzpdg"]
Sep 17 23:14:30.094043 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.094022    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.096235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.096207    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-94b6fc96d-x49tb"]
Sep 17 23:14:30.097808 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.097788    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.098547 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.098526    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hgwrz\""
Sep 17 23:14:30.098674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.098535    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 23:14:30.098674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.098651    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 23:14:30.098912 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.098892    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c8d5ffff4-xj6td"]
Sep 17 23:14:30.098964 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.098925    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.099035 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.099017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.102120 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.102096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.106007 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.105977    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 23:14:30.106114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 23:14:30.106265 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106245    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:14:30.106320 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106278    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 23:14:30.106320 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106257    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.106423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106365    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9dbhz\""
Sep 17 23:14:30.106481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 23:14:30.106783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106770    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-npjj2\""
Sep 17 23:14:30.106867 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106850    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.106902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.106879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:14:30.110943 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.110918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:14:30.117159 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.117141    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-hzpdg"]
Sep 17 23:14:30.117505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.117489    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 23:14:30.123055 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.123036    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:14:30.133252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.133223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-94b6fc96d-x49tb"]
Sep 17 23:14:30.146080 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.146051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8d5ffff4-xj6td"]
Sep 17 23:14:30.161691 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-default-certificate\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.161773 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.161773 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6btx\" (UniqueName: \"kubernetes.io/projected/b697b71a-4297-4445-9f49-092a0217e8b8-kube-api-access-c6btx\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.161843 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-trusted-ca\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.161843 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-stats-auth\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.161843 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db1763ab-9b54-4c5a-81b6-f390e9d0859b-config\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.161952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqmv\" (UniqueName: \"kubernetes.io/projected/db1763ab-9b54-4c5a-81b6-f390e9d0859b-kube-api-access-zcqmv\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.161952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161877    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db1763ab-9b54-4c5a-81b6-f390e9d0859b-trusted-ca\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.161952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-ca-trust-extracted\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.161952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-certificates\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.161952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-bound-sa-token\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.162114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.162114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.162114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.161996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-installation-pull-secrets\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.162114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.162015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqxqs\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-kube-api-access-qqxqs\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.162114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.162035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db1763ab-9b54-4c5a-81b6-f390e9d0859b-serving-cert\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.162114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.162051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-image-registry-private-configuration\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.186559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.186523    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"]
Sep 17 23:14:30.189274 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.189259    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"]
Sep 17 23:14:30.189377 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.189361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:30.191783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.191766    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"]
Sep 17 23:14:30.191897 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.191881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.192343 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.192326    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fn68g\""
Sep 17 23:14:30.192803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.192788    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.192885 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.192814    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 23:14:30.193049 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.193033    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.194452 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.194432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 23:14:30.194745 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.194728    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.194834 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.194748    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"]
Sep 17 23:14:30.194834 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.194763    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 23:14:30.194834 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.194794    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dnn76\""
Sep 17 23:14:30.195013 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.194882    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"
Sep 17 23:14:30.195128 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.195106    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.197168 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.197154    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2vwbr\""
Sep 17 23:14:30.197446 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.197432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.199845 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.199825    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.199943 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.199913    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 23:14:30.200012 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.199972    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.200111 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.200040    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tb9n8\""
Sep 17 23:14:30.200111 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.199975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 23:14:30.203403 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.203383    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"]
Sep 17 23:14:30.204144 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.204125    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"]
Sep 17 23:14:30.205003 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.204986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"]
Sep 17 23:14:30.218724 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.218699    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"]
Sep 17 23:14:30.262346 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.262427 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqxqs\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-kube-api-access-qqxqs\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.262427 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db1763ab-9b54-4c5a-81b6-f390e9d0859b-serving-cert\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.262427 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-image-registry-private-configuration\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.262526 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.262463    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:14:30.262526 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.262480    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8d5ffff4-xj6td: secret "image-registry-tls" not found
Sep 17 23:14:30.262526 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-config\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.262680 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.262539    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls podName:70b37b0f-b030-45ed-b87b-bf9f1d7a5b92 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:30.762521448 +0000 UTC m=+231.214352927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls") pod "image-registry-5c8d5ffff4-xj6td" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92") : secret "image-registry-tls" not found
Sep 17 23:14:30.262680 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5zv\" (UniqueName: \"kubernetes.io/projected/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-kube-api-access-jd5zv\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.262788 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-default-certificate\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.262788 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrkp\" (UniqueName: \"kubernetes.io/projected/bad94791-5a45-4383-bba5-64d08717cde3-kube-api-access-gsrkp\") pod \"network-check-source-7cbc45fddc-kn5mt\" (UID: \"bad94791-5a45-4383-bba5-64d08717cde3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"
Sep 17 23:14:30.262788 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-stats-auth\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.262788 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-serving-cert\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.262978 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db1763ab-9b54-4c5a-81b6-f390e9d0859b-config\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.262978 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6591d17a-2227-4fc6-bcc5-cdc82be2605e-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.262978 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-ca-trust-extracted\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.262978 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-certificates\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.262978 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-bound-sa-token\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.262978 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.262929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.263265 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.263012    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:14:30.263265 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.263067    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:30.763051049 +0000 UTC m=+231.214882535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : secret "router-metrics-certs-default" not found
Sep 17 23:14:30.263265 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-installation-pull-secrets\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.263265 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6591d17a-2227-4fc6-bcc5-cdc82be2605e-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.263265 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.263265 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6btx\" (UniqueName: \"kubernetes.io/projected/b697b71a-4297-4445-9f49-092a0217e8b8-kube-api-access-c6btx\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-trusted-ca\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflfq\" (UniqueName: \"kubernetes.io/projected/6591d17a-2227-4fc6-bcc5-cdc82be2605e-kube-api-access-kflfq\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqmv\" (UniqueName: \"kubernetes.io/projected/db1763ab-9b54-4c5a-81b6-f390e9d0859b-kube-api-access-zcqmv\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db1763ab-9b54-4c5a-81b6-f390e9d0859b-trusted-ca\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwrl\" (UniqueName: \"kubernetes.io/projected/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-kube-api-access-lpwrl\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db1763ab-9b54-4c5a-81b6-f390e9d0859b-config\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.263559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-certificates\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.264003 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.263722    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:30.763705622 +0000 UTC m=+231.215537119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:14:30.264003 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.263792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-ca-trust-extracted\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.264325 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.264298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db1763ab-9b54-4c5a-81b6-f390e9d0859b-trusted-ca\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.264632 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.264591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-trusted-ca\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.265109 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.265085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db1763ab-9b54-4c5a-81b6-f390e9d0859b-serving-cert\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.265284 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.265266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-stats-auth\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.265284 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.265274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-image-registry-private-configuration\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.265988 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.265973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-installation-pull-secrets\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.266282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.266268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-default-certificate\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.273148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.273125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqxqs\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-kube-api-access-qqxqs\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.274085 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.274062    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-bound-sa-token\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.274467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.274447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqmv\" (UniqueName: \"kubernetes.io/projected/db1763ab-9b54-4c5a-81b6-f390e9d0859b-kube-api-access-zcqmv\") pod \"console-operator-fd4bb46c-hzpdg\" (UID: \"db1763ab-9b54-4c5a-81b6-f390e9d0859b\") " pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.274540 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.274449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6btx\" (UniqueName: \"kubernetes.io/projected/b697b71a-4297-4445-9f49-092a0217e8b8-kube-api-access-c6btx\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.364145 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrkp\" (UniqueName: \"kubernetes.io/projected/bad94791-5a45-4383-bba5-64d08717cde3-kube-api-access-gsrkp\") pod \"network-check-source-7cbc45fddc-kn5mt\" (UID: \"bad94791-5a45-4383-bba5-64d08717cde3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"
Sep 17 23:14:30.364145 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-serving-cert\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.364145 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6591d17a-2227-4fc6-bcc5-cdc82be2605e-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.364354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6591d17a-2227-4fc6-bcc5-cdc82be2605e-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.364354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:30.364354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflfq\" (UniqueName: \"kubernetes.io/projected/6591d17a-2227-4fc6-bcc5-cdc82be2605e-kube-api-access-kflfq\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.364515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwrl\" (UniqueName: \"kubernetes.io/projected/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-kube-api-access-lpwrl\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:30.364515 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.364413    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:14:30.364515 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.364469    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls podName:f175cbfe-039c-418c-b8ac-5b01cf0f77e8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:30.864450526 +0000 UTC m=+231.316281993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zgdt7" (UID: "f175cbfe-039c-418c-b8ac-5b01cf0f77e8") : secret "samples-operator-tls" not found
Sep 17 23:14:30.364515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-config\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.364802 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5zv\" (UniqueName: \"kubernetes.io/projected/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-kube-api-access-jd5zv\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.364802 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6591d17a-2227-4fc6-bcc5-cdc82be2605e-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.364970 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.364950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-config\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.366002 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.365978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6591d17a-2227-4fc6-bcc5-cdc82be2605e-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.366328 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.366312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-serving-cert\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.372734 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.372708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrkp\" (UniqueName: \"kubernetes.io/projected/bad94791-5a45-4383-bba5-64d08717cde3-kube-api-access-gsrkp\") pod \"network-check-source-7cbc45fddc-kn5mt\" (UID: \"bad94791-5a45-4383-bba5-64d08717cde3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"
Sep 17 23:14:30.372829 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.372784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflfq\" (UniqueName: \"kubernetes.io/projected/6591d17a-2227-4fc6-bcc5-cdc82be2605e-kube-api-access-kflfq\") pod \"kube-storage-version-migrator-operator-55889f7b47-xwkcq\" (UID: \"6591d17a-2227-4fc6-bcc5-cdc82be2605e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.373134 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.373114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwrl\" (UniqueName: \"kubernetes.io/projected/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-kube-api-access-lpwrl\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:30.374087 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.374068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5zv\" (UniqueName: \"kubernetes.io/projected/ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b-kube-api-access-jd5zv\") pod \"service-ca-operator-57cdc4b768-5dqk5\" (UID: \"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.405019 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.404990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:30.505389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.505362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"
Sep 17 23:14:30.511281 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.511267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"
Sep 17 23:14:30.516013 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.515991    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"
Sep 17 23:14:30.672074 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.672052    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-hzpdg"]
Sep 17 23:14:30.674033 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:14:30.674010    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1763ab_9b54_4c5a_81b6_f390e9d0859b.slice/crio-037378c130b001a75fddad93a4839f5287477eb3baeb8ff6fd1c7285870f9710 WatchSource:0}: Error finding container 037378c130b001a75fddad93a4839f5287477eb3baeb8ff6fd1c7285870f9710: Status 404 returned error can't find the container with id 037378c130b001a75fddad93a4839f5287477eb3baeb8ff6fd1c7285870f9710
Sep 17 23:14:30.695332 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.695311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" event={"ID":"db1763ab-9b54-4c5a-81b6-f390e9d0859b","Type":"ContainerStarted","Data":"037378c130b001a75fddad93a4839f5287477eb3baeb8ff6fd1c7285870f9710"}
Sep 17 23:14:30.767698 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.767676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.767787 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.767731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:30.767787 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.767779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:30.767864 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.767822    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:31.76780551 +0000 UTC m=+232.219636976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:14:30.767913 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.767888    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:14:30.767913 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.767904    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8d5ffff4-xj6td: secret "image-registry-tls" not found
Sep 17 23:14:30.767985 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.767947    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls podName:70b37b0f-b030-45ed-b87b-bf9f1d7a5b92 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:31.767931431 +0000 UTC m=+232.219762918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls") pod "image-registry-5c8d5ffff4-xj6td" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92") : secret "image-registry-tls" not found
Sep 17 23:14:30.768024 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.767984    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:14:30.768024 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.768023    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:31.768012448 +0000 UTC m=+232.219843914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : secret "router-metrics-certs-default" not found
Sep 17 23:14:30.780122 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.780090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt"]
Sep 17 23:14:30.781225 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.781205    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5"]
Sep 17 23:14:30.783296 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:14:30.783251    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad94791_5a45_4383_bba5_64d08717cde3.slice/crio-4140d275cb51c258f31c44db9c893321e4698fb0e9319ac5120c6aa719f3ce2a WatchSource:0}: Error finding container 4140d275cb51c258f31c44db9c893321e4698fb0e9319ac5120c6aa719f3ce2a: Status 404 returned error can't find the container with id 4140d275cb51c258f31c44db9c893321e4698fb0e9319ac5120c6aa719f3ce2a
Sep 17 23:14:30.783844 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:14:30.783820    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0528cc_e40e_4841_bcd7_4cb9b0f5a50b.slice/crio-9308e4ecd471ad831c27d6a28818279a9db6bf36af6bdbd085a56484f1c10ccc WatchSource:0}: Error finding container 9308e4ecd471ad831c27d6a28818279a9db6bf36af6bdbd085a56484f1c10ccc: Status 404 returned error can't find the container with id 9308e4ecd471ad831c27d6a28818279a9db6bf36af6bdbd085a56484f1c10ccc
Sep 17 23:14:30.798495 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.798470    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq"]
Sep 17 23:14:30.801803 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:14:30.801779    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6591d17a_2227_4fc6_bcc5_cdc82be2605e.slice/crio-5c06096f51c117cf19d294454c47ffa5712151868d3a9ad26a6635de5f84a160 WatchSource:0}: Error finding container 5c06096f51c117cf19d294454c47ffa5712151868d3a9ad26a6635de5f84a160: Status 404 returned error can't find the container with id 5c06096f51c117cf19d294454c47ffa5712151868d3a9ad26a6635de5f84a160
Sep 17 23:14:30.868328 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:30.868303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:30.868476 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.868457    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:14:30.868536 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:30.868524    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls podName:f175cbfe-039c-418c-b8ac-5b01cf0f77e8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:31.868505904 +0000 UTC m=+232.320337372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zgdt7" (UID: "f175cbfe-039c-418c-b8ac-5b01cf0f77e8") : secret "samples-operator-tls" not found
Sep 17 23:14:31.704598 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.704023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt" event={"ID":"bad94791-5a45-4383-bba5-64d08717cde3","Type":"ContainerStarted","Data":"4bcb2defc9840cb93f7adf7c01a192f58cbb3f6a4cf8f3666716307f44060a41"}
Sep 17 23:14:31.704598 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.704079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt" event={"ID":"bad94791-5a45-4383-bba5-64d08717cde3","Type":"ContainerStarted","Data":"4140d275cb51c258f31c44db9c893321e4698fb0e9319ac5120c6aa719f3ce2a"}
Sep 17 23:14:31.706552 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.706481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5" event={"ID":"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b","Type":"ContainerStarted","Data":"9308e4ecd471ad831c27d6a28818279a9db6bf36af6bdbd085a56484f1c10ccc"}
Sep 17 23:14:31.708020 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.707980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq" event={"ID":"6591d17a-2227-4fc6-bcc5-cdc82be2605e","Type":"ContainerStarted","Data":"5c06096f51c117cf19d294454c47ffa5712151868d3a9ad26a6635de5f84a160"}
Sep 17 23:14:31.721674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.720858    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kn5mt" podStartSLOduration=1.720841666 podStartE2EDuration="1.720841666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:31.719959113 +0000 UTC m=+232.171790606" watchObservedRunningTime="2026-09-17 23:14:31.720841666 +0000 UTC m=+232.172673150"
Sep 17 23:14:31.775079 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.775052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:31.775200 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.775178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:31.775269 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.775235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:31.775402 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.775334    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:14:31.775402 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.775357    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8d5ffff4-xj6td: secret "image-registry-tls" not found
Sep 17 23:14:31.775402 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.775380    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:33.775362942 +0000 UTC m=+234.227194413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:14:31.775402 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.775407    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls podName:70b37b0f-b030-45ed-b87b-bf9f1d7a5b92 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:33.775390464 +0000 UTC m=+234.227221930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls") pod "image-registry-5c8d5ffff4-xj6td" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92") : secret "image-registry-tls" not found
Sep 17 23:14:31.775673 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.775468    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:14:31.775673 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.775507    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:33.775496839 +0000 UTC m=+234.227328311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : secret "router-metrics-certs-default" not found
Sep 17 23:14:31.875958 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:31.875920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:31.876103 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.876066    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:14:31.876167 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:31.876120    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls podName:f175cbfe-039c-418c-b8ac-5b01cf0f77e8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:33.876103706 +0000 UTC m=+234.327935187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zgdt7" (UID: "f175cbfe-039c-418c-b8ac-5b01cf0f77e8") : secret "samples-operator-tls" not found
Sep 17 23:14:33.773103 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.773081    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8cwbr"]
Sep 17 23:14:33.777558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.777538    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.780196 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.780048    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:14:33.780196 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.780051    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:14:33.780196 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.780191    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:14:33.781267 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.781251    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v429\""
Sep 17 23:14:33.781267 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.781263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:14:33.786134 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.786077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8cwbr"]
Sep 17 23:14:33.790713 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.790693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:33.790820 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.790722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-data-volume\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.790820 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.790758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-crio-socket\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.790820 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.790786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:33.790975 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.790831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.790975 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.790851    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:14:33.790975 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.790865    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:14:33.790975 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.790868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787g2\" (UniqueName: \"kubernetes.io/projected/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-api-access-787g2\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.790975 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.790870    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8d5ffff4-xj6td: secret "image-registry-tls" not found
Sep 17 23:14:33.790975 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.790912    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.790895936 +0000 UTC m=+238.242727420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : secret "router-metrics-certs-default" not found
Sep 17 23:14:33.790975 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.790978    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls podName:70b37b0f-b030-45ed-b87b-bf9f1d7a5b92 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.790960511 +0000 UTC m=+238.242791992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls") pod "image-registry-5c8d5ffff4-xj6td" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92") : secret "image-registry-tls" not found
Sep 17 23:14:33.791356 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.791025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:33.791356 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.791066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.791356 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.791167    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.791150382 +0000 UTC m=+238.242981871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:14:33.891790 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.891762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-data-volume\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.891897 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.891827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-crio-socket\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.891897 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.891856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.891897 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.891872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787g2\" (UniqueName: \"kubernetes.io/projected/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-api-access-787g2\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.892054 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.891956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-crio-socket\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.892054 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.891973    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:33.892054 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.892013    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:33.892054 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.892036    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:34.392018181 +0000 UTC m=+234.843849655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:33.892233 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.892068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.892233 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.892094    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:14:33.892233 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.892096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-data-volume\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:33.892233 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.892145    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls podName:f175cbfe-039c-418c-b8ac-5b01cf0f77e8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.892128059 +0000 UTC m=+238.343959539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zgdt7" (UID: "f175cbfe-039c-418c-b8ac-5b01cf0f77e8") : secret "samples-operator-tls" not found
Sep 17 23:14:33.892233 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.892172    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:33.892233 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:33.892205    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:34.392195411 +0000 UTC m=+234.844026877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:33.900333 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:33.900313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-787g2\" (UniqueName: \"kubernetes.io/projected/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-api-access-787g2\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:34.395516 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.395498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:34.395645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.395600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:34.395695 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:34.395662    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:34.395695 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:34.395684    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:34.395760 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:34.395718    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:35.395703167 +0000 UTC m=+235.847534636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:34.395760 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:34.395737    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:35.395726957 +0000 UTC m=+235.847558425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:34.720066 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.720009    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/0.log"
Sep 17 23:14:34.720066 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.720047    2566 generic.go:356] "Generic (PLEG): container finished" podID="db1763ab-9b54-4c5a-81b6-f390e9d0859b" containerID="70f7d0576a506eb7d51bc0d5f50fcc54be1fb17153ff804f8f22736bb878f5f2" exitCode=255
Sep 17 23:14:34.720232 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.720141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" event={"ID":"db1763ab-9b54-4c5a-81b6-f390e9d0859b","Type":"ContainerDied","Data":"70f7d0576a506eb7d51bc0d5f50fcc54be1fb17153ff804f8f22736bb878f5f2"}
Sep 17 23:14:34.720340 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.720322    2566 scope.go:117] "RemoveContainer" containerID="70f7d0576a506eb7d51bc0d5f50fcc54be1fb17153ff804f8f22736bb878f5f2"
Sep 17 23:14:34.721783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.721759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5" event={"ID":"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b","Type":"ContainerStarted","Data":"bf487791f2698dea89a06cc4d4e01216b63981d1cd6388643afb90a307c6bf2c"}
Sep 17 23:14:34.729070 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.729046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq" event={"ID":"6591d17a-2227-4fc6-bcc5-cdc82be2605e","Type":"ContainerStarted","Data":"a2fda411c90f11200d67127a09f39da3fda5e51ecb400d07b22e1e884cc7ef47"}
Sep 17 23:14:34.753904 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.753866    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq" podStartSLOduration=1.8257268899999999 podStartE2EDuration="4.753854739s" totalImagesPullingTime="2.928127849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:34.752941 +0000 UTC m=+235.204772487" watchObservedRunningTime="2026-09-17 23:14:34.753854739 +0000 UTC m=+235.205686227"
Sep 17 23:14:34.768359 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:34.768322    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5" podStartSLOduration=1.819111646 podStartE2EDuration="4.768308384s" totalImagesPullingTime="2.949196738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:34.766832788 +0000 UTC m=+235.218664288" watchObservedRunningTime="2026-09-17 23:14:34.768308384 +0000 UTC m=+235.220139873"
Sep 17 23:14:35.404232 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.404205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:35.404644 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.404301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:35.404644 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:35.404365    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:35.404644 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:35.404379    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:35.404644 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:35.404417    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.404402634 +0000 UTC m=+237.856234105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:35.404644 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:35.404439    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.40442198 +0000 UTC m=+237.856253448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:35.734275 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.734257    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/1.log"
Sep 17 23:14:35.734715 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.734700    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/0.log"
Sep 17 23:14:35.734788 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.734733    2566 generic.go:356] "Generic (PLEG): container finished" podID="db1763ab-9b54-4c5a-81b6-f390e9d0859b" containerID="6ab0e2091935859b75216a79fec5f1b9ee9b234a4fbe01ab39b9414b63a7388c" exitCode=255
Sep 17 23:14:35.734841 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.734813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" event={"ID":"db1763ab-9b54-4c5a-81b6-f390e9d0859b","Type":"ContainerDied","Data":"6ab0e2091935859b75216a79fec5f1b9ee9b234a4fbe01ab39b9414b63a7388c"}
Sep 17 23:14:35.734891 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.734848    2566 scope.go:117] "RemoveContainer" containerID="70f7d0576a506eb7d51bc0d5f50fcc54be1fb17153ff804f8f22736bb878f5f2"
Sep 17 23:14:35.735122 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.735103    2566 scope.go:117] "RemoveContainer" containerID="6ab0e2091935859b75216a79fec5f1b9ee9b234a4fbe01ab39b9414b63a7388c"
Sep 17 23:14:35.735297 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:35.735276    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-hzpdg_openshift-console-operator(db1763ab-9b54-4c5a-81b6-f390e9d0859b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" podUID="db1763ab-9b54-4c5a-81b6-f390e9d0859b"
Sep 17 23:14:35.923042 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:35.923020    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gq5q5_938d9893-bdce-4bd2-8898-0ab74c7fc325/dns-node-resolver/0.log"
Sep 17 23:14:36.526475 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:36.526449    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nvz6b_0be814cf-7b90-4d33-9f35-f73fdf631402/node-ca/0.log"
Sep 17 23:14:36.742215 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:36.742187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/1.log"
Sep 17 23:14:36.742589 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:36.742555    2566 scope.go:117] "RemoveContainer" containerID="6ab0e2091935859b75216a79fec5f1b9ee9b234a4fbe01ab39b9414b63a7388c"
Sep 17 23:14:36.742767 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:36.742745    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-hzpdg_openshift-console-operator(db1763ab-9b54-4c5a-81b6-f390e9d0859b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" podUID="db1763ab-9b54-4c5a-81b6-f390e9d0859b"
Sep 17 23:14:37.323415 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:37.323384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjz88_e92de406-eab7-42ca-a693-611ad02f1377/serve-healthcheck-canary/0.log"
Sep 17 23:14:37.420011 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:37.419980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:37.420139 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.420118    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:37.420195 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.420175    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:37.420195 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:37.420117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:37.420273 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.420179    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:41.420161667 +0000 UTC m=+241.871993139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:37.420273 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.420251    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:41.420231835 +0000 UTC m=+241.872063306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:37.823873 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:37.823843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:37.823901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:37.823943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.823988    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.824006    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8d5ffff4-xj6td: secret "image-registry-tls" not found
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.824058    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls podName:70b37b0f-b030-45ed-b87b-bf9f1d7a5b92 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:45.8240435 +0000 UTC m=+246.275874971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls") pod "image-registry-5c8d5ffff4-xj6td" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92") : secret "image-registry-tls" not found
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.824074    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.824095    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:45.824078965 +0000 UTC m=+246.275910431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:14:37.824221 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.824118    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:45.824107511 +0000 UTC m=+246.275938983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : secret "router-metrics-certs-default" not found
Sep 17 23:14:37.924809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:37.924782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:37.924928 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.924909    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:14:37.924977 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:37.924963    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls podName:f175cbfe-039c-418c-b8ac-5b01cf0f77e8 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:45.924950501 +0000 UTC m=+246.376781967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zgdt7" (UID: "f175cbfe-039c-418c-b8ac-5b01cf0f77e8") : secret "samples-operator-tls" not found
Sep 17 23:14:40.405085 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:40.405046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:40.405443 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:40.405105    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:14:40.405443 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:40.405416    2566 scope.go:117] "RemoveContainer" containerID="6ab0e2091935859b75216a79fec5f1b9ee9b234a4fbe01ab39b9414b63a7388c"
Sep 17 23:14:40.405614 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:40.405568    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-hzpdg_openshift-console-operator(db1763ab-9b54-4c5a-81b6-f390e9d0859b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" podUID="db1763ab-9b54-4c5a-81b6-f390e9d0859b"
Sep 17 23:14:41.455540 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:41.455503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:41.455948 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:41.455558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:41.455948 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:41.455630    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:41.455948 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:41.455710    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:49.455693519 +0000 UTC m=+249.907524986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:41.455948 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:41.455727    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:41.455948 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:41.455782    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:49.455766255 +0000 UTC m=+249.907597722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:45.891123 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:45.891071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:45.891717 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:45.891160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:45.891717 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:45.891210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:14:45.891717 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:45.891264    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:01.891236248 +0000 UTC m=+262.343067714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:14:45.891717 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:45.891314    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:14:45.891717 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:45.891378    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs podName:b697b71a-4297-4445-9f49-092a0217e8b8 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:01.89135994 +0000 UTC m=+262.343191464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs") pod "router-default-94b6fc96d-x49tb" (UID: "b697b71a-4297-4445-9f49-092a0217e8b8") : secret "router-metrics-certs-default" not found
Sep 17 23:14:45.893900 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:45.893879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"image-registry-5c8d5ffff4-xj6td\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") " pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:45.992041 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:45.992002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:45.994388 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:45.994357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f175cbfe-039c-418c-b8ac-5b01cf0f77e8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zgdt7\" (UID: \"f175cbfe-039c-418c-b8ac-5b01cf0f77e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:46.020123 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.020096    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9dbhz\""
Sep 17 23:14:46.028264 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.028242    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:46.102013 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.101992    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fn68g\""
Sep 17 23:14:46.109286 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.109269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"
Sep 17 23:14:46.302496 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.302465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8d5ffff4-xj6td"]
Sep 17 23:14:46.305923 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:14:46.305898    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b37b0f_b030_45ed_b87b_bf9f1d7a5b92.slice/crio-0bbff42dfb5a3660500ddcf0f6ec1e0b91bab9fc8559456dff3dae852b5baf38 WatchSource:0}: Error finding container 0bbff42dfb5a3660500ddcf0f6ec1e0b91bab9fc8559456dff3dae852b5baf38: Status 404 returned error can't find the container with id 0bbff42dfb5a3660500ddcf0f6ec1e0b91bab9fc8559456dff3dae852b5baf38
Sep 17 23:14:46.376134 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.376112    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7"]
Sep 17 23:14:46.775182 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.775150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7" event={"ID":"f175cbfe-039c-418c-b8ac-5b01cf0f77e8","Type":"ContainerStarted","Data":"469395a3a87b9b0b8336e0bf2161d26988f26db18198ca7b3bb19152fe847e19"}
Sep 17 23:14:46.776487 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.776464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" event={"ID":"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92","Type":"ContainerStarted","Data":"a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90"}
Sep 17 23:14:46.776643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.776506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" event={"ID":"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92","Type":"ContainerStarted","Data":"0bbff42dfb5a3660500ddcf0f6ec1e0b91bab9fc8559456dff3dae852b5baf38"}
Sep 17 23:14:46.776643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.776620    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:14:46.795418 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:46.795375    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" podStartSLOduration=16.795361982 podStartE2EDuration="16.795361982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:46.794663049 +0000 UTC m=+247.246494564" watchObservedRunningTime="2026-09-17 23:14:46.795361982 +0000 UTC m=+247.247193467"
Sep 17 23:14:48.784009 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:48.783976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7" event={"ID":"f175cbfe-039c-418c-b8ac-5b01cf0f77e8","Type":"ContainerStarted","Data":"d6a8b061390e6cf776119e5a53a0725ce92bc3836d37b8e7a47613f4105fbed1"}
Sep 17 23:14:48.784393 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:48.784036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7" event={"ID":"f175cbfe-039c-418c-b8ac-5b01cf0f77e8","Type":"ContainerStarted","Data":"3156e74077b930caeb453362f6d0517816625dcc69cc4a590176280630aa57f0"}
Sep 17 23:14:48.801159 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:48.801115    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zgdt7" podStartSLOduration=16.967540603 podStartE2EDuration="18.801099778s" totalImagesPullingTime="1.833559175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:48.800682729 +0000 UTC m=+249.252514216" watchObservedRunningTime="2026-09-17 23:14:48.801099778 +0000 UTC m=+249.252931266"
Sep 17 23:14:49.522318 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:49.522277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:49.522480 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:49.522389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:49.522480 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:49.522457    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:49.522558 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:49.522532    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls podName:2812488d-c0cc-4067-9d1a-ac432eb0b7e1 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:05.522514371 +0000 UTC m=+265.974345837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8cwbr" (UID: "2812488d-c0cc-4067-9d1a-ac432eb0b7e1") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:49.522925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:49.522896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:14:51.941171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:51.941138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:14:51.943297 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:51.943280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdb8ea08-57db-405c-ac91-9de219efe014-metrics-certs\") pod \"network-metrics-daemon-zd5lf\" (UID: \"bdb8ea08-57db-405c-ac91-9de219efe014\") " pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:14:52.008828 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.008799    2566 scope.go:117] "RemoveContainer" containerID="6ab0e2091935859b75216a79fec5f1b9ee9b234a4fbe01ab39b9414b63a7388c"
Sep 17 23:14:52.111933 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.111912    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mx9lw\""
Sep 17 23:14:52.120057 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.120040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zd5lf"
Sep 17 23:14:52.402359 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.402303    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zd5lf"]
Sep 17 23:14:52.405040 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:14:52.405013    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb8ea08_57db_405c_ac91_9de219efe014.slice/crio-43b39c2401548c318dfbfaea273e4719b7c784f05eca23bd22f9ac08f35e7e6e WatchSource:0}: Error finding container 43b39c2401548c318dfbfaea273e4719b7c784f05eca23bd22f9ac08f35e7e6e: Status 404 returned error can't find the container with id 43b39c2401548c318dfbfaea273e4719b7c784f05eca23bd22f9ac08f35e7e6e
Sep 17 23:14:52.797925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.797892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zd5lf" event={"ID":"bdb8ea08-57db-405c-ac91-9de219efe014","Type":"ContainerStarted","Data":"43b39c2401548c318dfbfaea273e4719b7c784f05eca23bd22f9ac08f35e7e6e"}
Sep 17 23:14:52.799487 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.799469    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:14:52.799898 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.799878    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/1.log"
Sep 17 23:14:52.799983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.799919    2566 generic.go:356] "Generic (PLEG): container finished" podID="db1763ab-9b54-4c5a-81b6-f390e9d0859b" containerID="7bbac66448b91a95db74d5250bc1466458e5bb82fca042c6d3b09d2b0fa735f0" exitCode=255
Sep 17 23:14:52.799983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.799955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" event={"ID":"db1763ab-9b54-4c5a-81b6-f390e9d0859b","Type":"ContainerDied","Data":"7bbac66448b91a95db74d5250bc1466458e5bb82fca042c6d3b09d2b0fa735f0"}
Sep 17 23:14:52.800090 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.800005    2566 scope.go:117] "RemoveContainer" containerID="6ab0e2091935859b75216a79fec5f1b9ee9b234a4fbe01ab39b9414b63a7388c"
Sep 17 23:14:52.800339 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:52.800311    2566 scope.go:117] "RemoveContainer" containerID="7bbac66448b91a95db74d5250bc1466458e5bb82fca042c6d3b09d2b0fa735f0"
Sep 17 23:14:52.800516 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:14:52.800489    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-hzpdg_openshift-console-operator(db1763ab-9b54-4c5a-81b6-f390e9d0859b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" podUID="db1763ab-9b54-4c5a-81b6-f390e9d0859b"
Sep 17 23:14:53.805963 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:53.805933    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:14:53.807988 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:53.807954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zd5lf" event={"ID":"bdb8ea08-57db-405c-ac91-9de219efe014","Type":"ContainerStarted","Data":"c4ac22a655a772e2ae23fce26459fb5de72484cebd32812428a26216f54a8327"}
Sep 17 23:14:53.808080 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:53.808019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zd5lf" event={"ID":"bdb8ea08-57db-405c-ac91-9de219efe014","Type":"ContainerStarted","Data":"c8365b36835c68edf100bdf850a2695fb8200274c1e8a64469326fd36fa3cfdb"}
Sep 17 23:14:53.825584 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:53.825531    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zd5lf" podStartSLOduration=252.89354722 podStartE2EDuration="4m13.825515354s" totalImagesPullingTime="931.968134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:53.824017464 +0000 UTC m=+254.275848974" watchObservedRunningTime="2026-09-17 23:14:53.825515354 +0000 UTC m=+254.277346841"
Sep 17 23:14:59.602476 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:14:59.602436    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8d5ffff4-xj6td"]
Sep 17 23:15:00.405634 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:00.405598    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:15:00.405813 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:00.405652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:15:00.405942 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:00.405929    2566 scope.go:117] "RemoveContainer" containerID="7bbac66448b91a95db74d5250bc1466458e5bb82fca042c6d3b09d2b0fa735f0"
Sep 17 23:15:00.406118 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:00.406097    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-hzpdg_openshift-console-operator(db1763ab-9b54-4c5a-81b6-f390e9d0859b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" podUID="db1763ab-9b54-4c5a-81b6-f390e9d0859b"
Sep 17 23:15:01.920637 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:01.920563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:01.920637 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:01.920651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:01.921260 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:01.921238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b697b71a-4297-4445-9f49-092a0217e8b8-service-ca-bundle\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:01.923078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:01.923055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b697b71a-4297-4445-9f49-092a0217e8b8-metrics-certs\") pod \"router-default-94b6fc96d-x49tb\" (UID: \"b697b71a-4297-4445-9f49-092a0217e8b8\") " pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:02.214036 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:02.213968    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-npjj2\""
Sep 17 23:15:02.222319 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:02.222301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:02.496710 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:02.496676    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-94b6fc96d-x49tb"]
Sep 17 23:15:02.499705 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:15:02.499680    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb697b71a_4297_4445_9f49_092a0217e8b8.slice/crio-38c6ca398a0c367456656ddc98816e4ee859845cc6a621476760a3b2d2a1fe0e WatchSource:0}: Error finding container 38c6ca398a0c367456656ddc98816e4ee859845cc6a621476760a3b2d2a1fe0e: Status 404 returned error can't find the container with id 38c6ca398a0c367456656ddc98816e4ee859845cc6a621476760a3b2d2a1fe0e
Sep 17 23:15:02.841324 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:02.841243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-94b6fc96d-x49tb" event={"ID":"b697b71a-4297-4445-9f49-092a0217e8b8","Type":"ContainerStarted","Data":"8def1338ca6d4990111683981565432f26ae353fb178ea970fcdd019aaad77ac"}
Sep 17 23:15:02.841324 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:02.841307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-94b6fc96d-x49tb" event={"ID":"b697b71a-4297-4445-9f49-092a0217e8b8","Type":"ContainerStarted","Data":"38c6ca398a0c367456656ddc98816e4ee859845cc6a621476760a3b2d2a1fe0e"}
Sep 17 23:15:02.862060 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:02.861977    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-94b6fc96d-x49tb" podStartSLOduration=32.86196164 podStartE2EDuration="32.86196164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:02.860607674 +0000 UTC m=+263.312439159" watchObservedRunningTime="2026-09-17 23:15:02.86196164 +0000 UTC m=+263.313793128"
Sep 17 23:15:03.222810 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:03.222741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:03.225593 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:03.225551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:03.844819 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:03.844791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:03.846177 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:03.846152    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-94b6fc96d-x49tb"
Sep 17 23:15:05.546210 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:05.546170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:15:05.548738 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:05.548715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2812488d-c0cc-4067-9d1a-ac432eb0b7e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8cwbr\" (UID: \"2812488d-c0cc-4067-9d1a-ac432eb0b7e1\") " pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:15:05.607519 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:05.607491    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v429\""
Sep 17 23:15:05.615080 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:05.615062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8cwbr"
Sep 17 23:15:05.889483 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:05.889460    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8cwbr"]
Sep 17 23:15:05.891685 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:15:05.891666    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2812488d_c0cc_4067_9d1a_ac432eb0b7e1.slice/crio-10d057f28a8a6bbbf13c2f45ab4e1ff535b81b7a60126800d0fa4601e6c71f54 WatchSource:0}: Error finding container 10d057f28a8a6bbbf13c2f45ab4e1ff535b81b7a60126800d0fa4601e6c71f54: Status 404 returned error can't find the container with id 10d057f28a8a6bbbf13c2f45ab4e1ff535b81b7a60126800d0fa4601e6c71f54
Sep 17 23:15:06.856031 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:06.855958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8cwbr" event={"ID":"2812488d-c0cc-4067-9d1a-ac432eb0b7e1","Type":"ContainerStarted","Data":"a9992ea008e5012b22535e35a834dcd3ccf06c7a1f04a1bd2c88cd058c4d47e0"}
Sep 17 23:15:06.856031 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:06.856023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8cwbr" event={"ID":"2812488d-c0cc-4067-9d1a-ac432eb0b7e1","Type":"ContainerStarted","Data":"e220c830025e4bdd374c33842c7bc214ee096f028a2e2027fae3cab247612002"}
Sep 17 23:15:06.856386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:06.856053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8cwbr" event={"ID":"2812488d-c0cc-4067-9d1a-ac432eb0b7e1","Type":"ContainerStarted","Data":"10d057f28a8a6bbbf13c2f45ab4e1ff535b81b7a60126800d0fa4601e6c71f54"}
Sep 17 23:15:08.866041 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:08.866005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8cwbr" event={"ID":"2812488d-c0cc-4067-9d1a-ac432eb0b7e1","Type":"ContainerStarted","Data":"a7088cefd8177ea10caf2e1e9187a9d1df382ed02760f32178f0864467a6a3d1"}
Sep 17 23:15:08.884733 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:08.884688    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8cwbr" podStartSLOduration=33.974149705 podStartE2EDuration="35.884674102s" totalImagesPullingTime="1.910524397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:14:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:08.883625677 +0000 UTC m=+269.335457166" watchObservedRunningTime="2026-09-17 23:15:08.884674102 +0000 UTC m=+269.336505584"
Sep 17 23:15:09.607964 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:09.607923    2566 patch_prober.go:28] interesting pod/image-registry-5c8d5ffff4-xj6td container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 23:15:09.608161 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:09.607987    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" podUID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:15:11.008328 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:11.008299    2566 scope.go:117] "RemoveContainer" containerID="7bbac66448b91a95db74d5250bc1466458e5bb82fca042c6d3b09d2b0fa735f0"
Sep 17 23:15:11.008719 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:11.008445    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-hzpdg_openshift-console-operator(db1763ab-9b54-4c5a-81b6-f390e9d0859b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" podUID="db1763ab-9b54-4c5a-81b6-f390e9d0859b"
Sep 17 23:15:12.305450 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.305413    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-zndfk"]
Sep 17 23:15:12.308914 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.308890    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.311760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.311731    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:15:12.312023 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.312003    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:15:12.312121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.312052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 23:15:12.312121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.312066    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:15:12.313381 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.313358    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2rtg9\""
Sep 17 23:15:12.313497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.313359    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:15:12.313497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.313476    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 23:15:12.324862 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.324839    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fpd8l"]
Sep 17 23:15:12.328020 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.328001    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-zndfk"]
Sep 17 23:15:12.328109 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.328093    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.330456 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.330434    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bbsjl\""
Sep 17 23:15:12.330456 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.330434    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:15:12.330711 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.330696    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:15:12.330793 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.330744    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:15:12.392347 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.392323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.392450 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.392352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h74w\" (UniqueName: \"kubernetes.io/projected/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-api-access-5h74w\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.392450 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.392372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.392450 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.392395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.392634 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.392474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d3df576-687c-4581-ad5a-09f859fd41c2-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.392634 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.392531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d3df576-687c-4581-ad5a-09f859fd41c2-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.493531 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d3df576-687c-4581-ad5a-09f859fd41c2-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.493663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-tls\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.493663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.493663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d3df576-687c-4581-ad5a-09f859fd41c2-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.493832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-sys\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.493832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xl2\" (UniqueName: \"kubernetes.io/projected/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-kube-api-access-n5xl2\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.493832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-root\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.493832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-textfile\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.493832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.493832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-metrics-client-ca\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h74w\" (UniqueName: \"kubernetes.io/projected/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-api-access-5h74w\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:12.493900    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-wtmp\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:12.493960    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-tls podName:8d3df576-687c-4581-ad5a-09f859fd41c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:12.993939993 +0000 UTC m=+273.445771464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-zndfk" (UID: "8d3df576-687c-4581-ad5a-09f859fd41c2") : secret "kube-state-metrics-tls" not found
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.493981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d3df576-687c-4581-ad5a-09f859fd41c2-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.494077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.494020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.494331 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.494130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d3df576-687c-4581-ad5a-09f859fd41c2-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.494485 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.494458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.496514 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.496483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.505225 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.505201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h74w\" (UniqueName: \"kubernetes.io/projected/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-api-access-5h74w\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:12.595247 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-sys\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xl2\" (UniqueName: \"kubernetes.io/projected/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-kube-api-access-n5xl2\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-root\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-textfile\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-sys\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-root\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-metrics-client-ca\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-wtmp\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-tls\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595809 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:12.595628    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:15:12.595809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-wtmp\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-textfile\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595809 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:12.595704    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-tls podName:271151eb-ca8a-4d11-8b92-a1c3b30c5e31 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:13.095684345 +0000 UTC m=+273.547516029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-tls") pod "node-exporter-fpd8l" (UID: "271151eb-ca8a-4d11-8b92-a1c3b30c5e31") : secret "node-exporter-tls" not found
Sep 17 23:15:12.595809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.595979 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.595948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-metrics-client-ca\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.597327 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.597311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.603563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.603545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xl2\" (UniqueName: \"kubernetes.io/projected/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-kube-api-access-n5xl2\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:12.998525 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:12.998503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:13.000739 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.000721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d3df576-687c-4581-ad5a-09f859fd41c2-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zndfk\" (UID: \"8d3df576-687c-4581-ad5a-09f859fd41c2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:13.099193 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.099167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-tls\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:13.101248 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.101222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/271151eb-ca8a-4d11-8b92-a1c3b30c5e31-node-exporter-tls\") pod \"node-exporter-fpd8l\" (UID: \"271151eb-ca8a-4d11-8b92-a1c3b30c5e31\") " pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:13.218782 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.218757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk"
Sep 17 23:15:13.236756 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.236733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fpd8l"
Sep 17 23:15:13.249783 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:15:13.249706    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271151eb_ca8a_4d11_8b92_a1c3b30c5e31.slice/crio-d8d6d75ae3782a515ab5766a10702a90082cc507dc3a02a8c3318aff0d1ff4e7 WatchSource:0}: Error finding container d8d6d75ae3782a515ab5766a10702a90082cc507dc3a02a8c3318aff0d1ff4e7: Status 404 returned error can't find the container with id d8d6d75ae3782a515ab5766a10702a90082cc507dc3a02a8c3318aff0d1ff4e7
Sep 17 23:15:13.498295 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.498255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-zndfk"]
Sep 17 23:15:13.502570 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:15:13.502540    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3df576_687c_4581_ad5a_09f859fd41c2.slice/crio-be87d0ac03c29c57bebfe978c2fdbcb62307e401f882a6f3f93c284f751f2a1a WatchSource:0}: Error finding container be87d0ac03c29c57bebfe978c2fdbcb62307e401f882a6f3f93c284f751f2a1a: Status 404 returned error can't find the container with id be87d0ac03c29c57bebfe978c2fdbcb62307e401f882a6f3f93c284f751f2a1a
Sep 17 23:15:13.886375 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.886297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpd8l" event={"ID":"271151eb-ca8a-4d11-8b92-a1c3b30c5e31","Type":"ContainerStarted","Data":"d8d6d75ae3782a515ab5766a10702a90082cc507dc3a02a8c3318aff0d1ff4e7"}
Sep 17 23:15:13.887534 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:13.887511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk" event={"ID":"8d3df576-687c-4581-ad5a-09f859fd41c2","Type":"ContainerStarted","Data":"be87d0ac03c29c57bebfe978c2fdbcb62307e401f882a6f3f93c284f751f2a1a"}
Sep 17 23:15:14.894129 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:14.894092    2566 generic.go:356] "Generic (PLEG): container finished" podID="271151eb-ca8a-4d11-8b92-a1c3b30c5e31" containerID="3e28a42bfd91b8c9c0bf5e2c35c853f3f3456d2a66851bed087f461b3a0bb2b8" exitCode=0
Sep 17 23:15:14.894505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:14.894211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpd8l" event={"ID":"271151eb-ca8a-4d11-8b92-a1c3b30c5e31","Type":"ContainerDied","Data":"3e28a42bfd91b8c9c0bf5e2c35c853f3f3456d2a66851bed087f461b3a0bb2b8"}
Sep 17 23:15:14.898137 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:14.898105    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk" event={"ID":"8d3df576-687c-4581-ad5a-09f859fd41c2","Type":"ContainerStarted","Data":"b42a7fdbd396aacfcbb6dba8481c466eafed8b5c86344a8fa76c74258592e673"}
Sep 17 23:15:14.898238 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:14.898170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk" event={"ID":"8d3df576-687c-4581-ad5a-09f859fd41c2","Type":"ContainerStarted","Data":"800c52a89c3aec1e760e17560d5a25378fb2a55fa5c0d13032a8f87976d3f3ee"}
Sep 17 23:15:15.905341 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:15.905298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpd8l" event={"ID":"271151eb-ca8a-4d11-8b92-a1c3b30c5e31","Type":"ContainerStarted","Data":"76208f022e9fc9d18f197b60c3085707c2f119b26d61b7e948def49947377ff6"}
Sep 17 23:15:15.905832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:15.905385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpd8l" event={"ID":"271151eb-ca8a-4d11-8b92-a1c3b30c5e31","Type":"ContainerStarted","Data":"8ba40ba7e028b0836ce127dbf9e92fc56d3cbfb2b6b03eda55c7a598ae51fdcb"}
Sep 17 23:15:15.907949 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:15.907921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk" event={"ID":"8d3df576-687c-4581-ad5a-09f859fd41c2","Type":"ContainerStarted","Data":"d1e1c782fd9f81aa169a5a2859462a4f832d6da06c7b308ae88c5738c955ac7f"}
Sep 17 23:15:15.926535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:15.926482    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fpd8l" podStartSLOduration=2.895693643 podStartE2EDuration="3.92646674s" totalImagesPullingTime="960.564158ms" totalInitContainerRuntime="70.208939ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:15.923874141 +0000 UTC m=+276.375705629" watchObservedRunningTime="2026-09-17 23:15:15.92646674 +0000 UTC m=+276.378298228"
Sep 17 23:15:15.940779 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:15.940729    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-zndfk" podStartSLOduration=2.672997631 podStartE2EDuration="3.940716379s" totalImagesPullingTime="1.267718748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:15.94014747 +0000 UTC m=+276.391978959" watchObservedRunningTime="2026-09-17 23:15:15.940716379 +0000 UTC m=+276.392547866"
Sep 17 23:15:17.497011 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.496972    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"]
Sep 17 23:15:17.500514 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.500493    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.503225 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.503196    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 23:15:17.503357 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.503275    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 23:15:17.503357 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.503342    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 23:15:17.503463 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.503369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 23:15:17.503555 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.503534    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 23:15:17.503663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.503534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fspfl\""
Sep 17 23:15:17.508718 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.508693    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 23:15:17.514431 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.514408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"]
Sep 17 23:15:17.632809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.632788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.632907 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.632822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzqg\" (UniqueName: \"kubernetes.io/projected/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-kube-api-access-rrzqg\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.632907 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.632848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-metrics-client-ca\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.632975 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.632943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-telemeter-client-tls\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.633010 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.632978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.633045 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.633029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-federate-client-tls\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.633078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.633047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-serving-certs-ca-bundle\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.633078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.633070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-secret-telemeter-client\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734372 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzqg\" (UniqueName: \"kubernetes.io/projected/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-kube-api-access-rrzqg\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734372 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-metrics-client-ca\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734372 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-telemeter-client-tls\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734480 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734480 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-federate-client-tls\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734480 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-serving-certs-ca-bundle\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.734682 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.734637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-secret-telemeter-client\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.735356 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.735291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-serving-certs-ca-bundle\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.735356 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.735291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-metrics-client-ca\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.735762 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.735729    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.737475 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.737448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.737930 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.737905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-secret-telemeter-client\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.737989 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.737905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-federate-client-tls\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.738027 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.737993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-telemeter-client-tls\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.742214 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.742185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzqg\" (UniqueName: \"kubernetes.io/projected/ede67b21-37b8-48cc-9b38-c2d30dccbfa5-kube-api-access-rrzqg\") pod \"telemeter-client-76b5dd45c-7zf7w\" (UID: \"ede67b21-37b8-48cc-9b38-c2d30dccbfa5\") " pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:17.810181 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:17.810136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"
Sep 17 23:15:18.084546 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.084457    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76b5dd45c-7zf7w"]
Sep 17 23:15:18.088272 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:15:18.088246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede67b21_37b8_48cc_9b38_c2d30dccbfa5.slice/crio-716f2ae7edd8702579108a7146460661ed177462d0ebf0e48c39b70d98c79a9e WatchSource:0}: Error finding container 716f2ae7edd8702579108a7146460661ed177462d0ebf0e48c39b70d98c79a9e: Status 404 returned error can't find the container with id 716f2ae7edd8702579108a7146460661ed177462d0ebf0e48c39b70d98c79a9e
Sep 17 23:15:18.484272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.484247    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:15:18.489323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.489305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.492048 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.492021    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:15:18.492615 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.492569    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:15:18.492615 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.492616    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:15:18.492778 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.492621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:15:18.492778 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.492717    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:15:18.493558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493362    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:15:18.493558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493379    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2d92w\""
Sep 17 23:15:18.493558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:15:18.493558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493434    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:15:18.493558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493488    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:15:18.493558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493379    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:15:18.493558 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493478    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:15:18.493962 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493361    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4dall3dgth676\""
Sep 17 23:15:18.493962 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.493896    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:15:18.494719 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.494700    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:15:18.494806 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.494720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:15:18.496011 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.495993    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:15:18.502737 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.502713    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:15:18.641589 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-config-out\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641910 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-config\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641910 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641910 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-web-config\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641910 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.641910 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.641981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642052    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9c2\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-kube-api-access-gr9c2\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.642479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.642232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-config-out\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743709 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743709 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743709 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-config\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743709 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743709 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-web-config\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.743709 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.743953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9c2\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-kube-api-access-gr9c2\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744005 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.744000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744546 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.744173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.744546 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.744536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.746199 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.745087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.746199 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.745831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.746463 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.746379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-config-out\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.746750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.746780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.746797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.746986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747340 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.747179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747408 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.747387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747813 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.747752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.747898 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.747809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.748616 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.748593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-config\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.748714 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.748685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-web-config\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.749275 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.749244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.749368 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.749277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.749483 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.749463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.749642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.749626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.754473 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.754456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9c2\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-kube-api-access-gr9c2\") pod \"prometheus-k8s-0\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.799552 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.799528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:18.923463 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:18.923427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w" event={"ID":"ede67b21-37b8-48cc-9b38-c2d30dccbfa5","Type":"ContainerStarted","Data":"716f2ae7edd8702579108a7146460661ed177462d0ebf0e48c39b70d98c79a9e"}
Sep 17 23:15:19.087451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:19.087394    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:15:19.089331 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:15:19.089307    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced819dd_27b0_4c70_b3f4_5992ff109965.slice/crio-a3f4c29ec4e3d5f71d64baa22d1f3953942f7946c466667627eea6263fced109 WatchSource:0}: Error finding container a3f4c29ec4e3d5f71d64baa22d1f3953942f7946c466667627eea6263fced109: Status 404 returned error can't find the container with id a3f4c29ec4e3d5f71d64baa22d1f3953942f7946c466667627eea6263fced109
Sep 17 23:15:19.609112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:19.609062    2566 patch_prober.go:28] interesting pod/image-registry-5c8d5ffff4-xj6td container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 23:15:19.609564 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:19.609126    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" podUID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:15:19.929477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:19.929436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w" event={"ID":"ede67b21-37b8-48cc-9b38-c2d30dccbfa5","Type":"ContainerStarted","Data":"4cc06dd2b06c82b950d6dd054b063047117a4a6b609ddb02a7f26842830b9d5d"}
Sep 17 23:15:19.930811 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:19.930789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerStarted","Data":"a3f4c29ec4e3d5f71d64baa22d1f3953942f7946c466667627eea6263fced109"}
Sep 17 23:15:20.516374 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:20.516346    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zzxvm" podUID="6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d"
Sep 17 23:15:20.937100 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:20.937065    2566 generic.go:356] "Generic (PLEG): container finished" podID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1" exitCode=0
Sep 17 23:15:20.937460 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:20.937152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"}
Sep 17 23:15:20.939817 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:20.939786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w" event={"ID":"ede67b21-37b8-48cc-9b38-c2d30dccbfa5","Type":"ContainerStarted","Data":"153cf6e1d6443aa47da2181cf78b261c7057405a6ce7d64c81eadec339524974"}
Sep 17 23:15:20.939928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:20.939821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:15:20.939928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:20.939842    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w" event={"ID":"ede67b21-37b8-48cc-9b38-c2d30dccbfa5","Type":"ContainerStarted","Data":"6f9df9585cb0a112e8b7563ea087bfda895d2542a98c6983f37223314c8605eb"}
Sep 17 23:15:20.983927 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:20.983883    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76b5dd45c-7zf7w" podStartSLOduration=1.693820568 podStartE2EDuration="3.983870843s" totalImagesPullingTime="2.290050275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:20.982613578 +0000 UTC m=+281.434445058" watchObservedRunningTime="2026-09-17 23:15:20.983870843 +0000 UTC m=+281.435702330"
Sep 17 23:15:24.392263 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.392166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:15:24.395093 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.395064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d-metrics-tls\") pod \"dns-default-zzxvm\" (UID: \"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d\") " pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:15:24.543796 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.543772    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvr9f\""
Sep 17 23:15:24.551083 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.551064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:15:24.622151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.622099    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" podUID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" containerName="registry" containerID="cri-o://a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90" gracePeriod=30
Sep 17 23:15:24.820127 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.820041    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzxvm"]
Sep 17 23:15:24.822633 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:15:24.822606    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec0d4f1_7308_4faa_aa7f_ce57dcc7690d.slice/crio-a8656de9b1e30e95acd5d0ca82ad0123f331c64384bde156213c41541b353718 WatchSource:0}: Error finding container a8656de9b1e30e95acd5d0ca82ad0123f331c64384bde156213c41541b353718: Status 404 returned error can't find the container with id a8656de9b1e30e95acd5d0ca82ad0123f331c64384bde156213c41541b353718
Sep 17 23:15:24.862209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.862189    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:15:24.958124 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.958093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzxvm" event={"ID":"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d","Type":"ContainerStarted","Data":"a8656de9b1e30e95acd5d0ca82ad0123f331c64384bde156213c41541b353718"}
Sep 17 23:15:24.959459 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.959421    2566 generic.go:356] "Generic (PLEG): container finished" podID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" containerID="a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90" exitCode=0
Sep 17 23:15:24.959629 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.959481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td"
Sep 17 23:15:24.959629 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.959498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" event={"ID":"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92","Type":"ContainerDied","Data":"a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90"}
Sep 17 23:15:24.959629 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.959537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8d5ffff4-xj6td" event={"ID":"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92","Type":"ContainerDied","Data":"0bbff42dfb5a3660500ddcf0f6ec1e0b91bab9fc8559456dff3dae852b5baf38"}
Sep 17 23:15:24.959629 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.959560    2566 scope.go:117] "RemoveContainer" containerID="a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90"
Sep 17 23:15:24.962148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.962124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerStarted","Data":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"}
Sep 17 23:15:24.962241 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.962191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerStarted","Data":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"}
Sep 17 23:15:24.970263 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.970187    2566 scope.go:117] "RemoveContainer" containerID="a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90"
Sep 17 23:15:24.970674 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:15:24.970648    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90\": container with ID starting with a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90 not found: ID does not exist" containerID="a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90"
Sep 17 23:15:24.970762 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.970686    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90"} err="failed to get container status \"a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90\": rpc error: code = NotFound desc = could not find container \"a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90\": container with ID starting with a98630d54fe0c213defa49ddd7638a7d3b389aed551e97c94941202c6eeb8f90 not found: ID does not exist"
Sep 17 23:15:24.997206 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997184    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-ca-trust-extracted\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.997308 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-image-registry-private-configuration\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.997308 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.997308 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-installation-pull-secrets\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.997466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqxqs\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-kube-api-access-qqxqs\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.997466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-bound-sa-token\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.997466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-trusted-ca\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.997466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.997424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-certificates\") pod \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\" (UID: \"70b37b0f-b030-45ed-b87b-bf9f1d7a5b92\") "
Sep 17 23:15:24.998115 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.998003    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:15:24.998115 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:24.998069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:15:25.000600 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.000563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:15:25.000751 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.000732    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:15:25.000861 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.000839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:15:25.000940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.000923    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-kube-api-access-qqxqs" (OuterVolumeSpecName: "kube-api-access-qqxqs") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "kube-api-access-qqxqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:15:25.001177 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.001150    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:15:25.007095 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.007072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" (UID: "70b37b0f-b030-45ed-b87b-bf9f1d7a5b92"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:15:25.008733 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.008717    2566 scope.go:117] "RemoveContainer" containerID="7bbac66448b91a95db74d5250bc1466458e5bb82fca042c6d3b09d2b0fa735f0"
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098496    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-certificates\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098527    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-ca-trust-extracted\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098542    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-image-registry-private-configuration\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098557    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-registry-tls\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098570    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-installation-pull-secrets\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098598    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqxqs\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-kube-api-access-qqxqs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098612    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-bound-sa-token\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.098645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.098625    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92-trusted-ca\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:15:25.353135 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.353109    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8d5ffff4-xj6td"]
Sep 17 23:15:25.358794 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.358411    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c8d5ffff4-xj6td"]
Sep 17 23:15:25.967718 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.967680    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc5e5836-d228-4823-bdba-f6cff18aa1d3" containerID="bad7c0998acb9b97186ad8d729842b9f4538427481439f6f7a889f0207da0c8b" exitCode=255
Sep 17 23:15:25.968231 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.967744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw" event={"ID":"fc5e5836-d228-4823-bdba-f6cff18aa1d3","Type":"ContainerDied","Data":"bad7c0998acb9b97186ad8d729842b9f4538427481439f6f7a889f0207da0c8b"}
Sep 17 23:15:25.969918 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.969893    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:15:25.970036 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.969995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" event={"ID":"db1763ab-9b54-4c5a-81b6-f390e9d0859b","Type":"ContainerStarted","Data":"d7c008431b6247a4ea45c99bf7672e57926a5bccf1ce817efefc829e6968186d"}
Sep 17 23:15:25.970505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.970485    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:15:25.975489 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.975470    2566 scope.go:117] "RemoveContainer" containerID="bad7c0998acb9b97186ad8d729842b9f4538427481439f6f7a889f0207da0c8b"
Sep 17 23:15:25.975827 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:25.975804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg"
Sep 17 23:15:26.013745 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.013724    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" path="/var/lib/kubelet/pods/70b37b0f-b030-45ed-b87b-bf9f1d7a5b92/volumes"
Sep 17 23:15:26.016366 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.016320    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-hzpdg" podStartSLOduration=52.963448447 podStartE2EDuration="56.01630402s" totalImagesPullingTime="3.052855573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:26.014506993 +0000 UTC m=+286.466338482" watchObservedRunningTime="2026-09-17 23:15:26.01630402 +0000 UTC m=+286.468135510"
Sep 17 23:15:26.977045 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.977017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzxvm" event={"ID":"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d","Type":"ContainerStarted","Data":"2eb021c8d9846b19622d5360f5730e1f3aa027d1737db2b1554b03e8c4572675"}
Sep 17 23:15:26.977440 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.977068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzxvm" event={"ID":"6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d","Type":"ContainerStarted","Data":"4842708a91ae29c3b77ac76f1b02490938fb19c8fa4dcb04dcffd0b68f009897"}
Sep 17 23:15:26.977440 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.977099    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:15:26.978967 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.978940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d8dd86b6-lw9mw" event={"ID":"fc5e5836-d228-4823-bdba-f6cff18aa1d3","Type":"ContainerStarted","Data":"78659ab87f4d5227de14f58948906de38115984bd222a72d64ed96d6946705e3"}
Sep 17 23:15:26.982546 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.982527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerStarted","Data":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"}
Sep 17 23:15:26.982662 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.982601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerStarted","Data":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"}
Sep 17 23:15:26.982662 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.982633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerStarted","Data":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"}
Sep 17 23:15:26.982662 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.982657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerStarted","Data":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"}
Sep 17 23:15:26.995598 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:26.995534    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zzxvm" podStartSLOduration=251.183924051 podStartE2EDuration="4m12.995522742s" totalImagesPullingTime="1.811598691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:26.99436029 +0000 UTC m=+287.446191780" watchObservedRunningTime="2026-09-17 23:15:26.995522742 +0000 UTC m=+287.447354229"
Sep 17 23:15:28.800212 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:28.800170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:36.990374 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:36.990340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zzxvm"
Sep 17 23:15:37.006840 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:37.006782    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=12.072283139 podStartE2EDuration="19.006762099s" totalImagesPullingTime="6.911396665s" totalInitContainerRuntime="23.082295ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:27.052088001 +0000 UTC m=+287.503919495" watchObservedRunningTime="2026-09-17 23:15:37.006762099 +0000 UTC m=+297.458593587"
Sep 17 23:15:39.958425 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:39.958169    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:15:39.959147 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:39.959121    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:15:39.967297 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:39.967277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:15:39.968153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:39.968135    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:15:39.970481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:39.970463    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:15:50.072668 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:50.072636    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b" containerID="bf487791f2698dea89a06cc4d4e01216b63981d1cd6388643afb90a307c6bf2c" exitCode=0
Sep 17 23:15:50.073047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:50.072707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5" event={"ID":"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b","Type":"ContainerDied","Data":"bf487791f2698dea89a06cc4d4e01216b63981d1cd6388643afb90a307c6bf2c"}
Sep 17 23:15:50.073047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:50.073028    2566 scope.go:117] "RemoveContainer" containerID="bf487791f2698dea89a06cc4d4e01216b63981d1cd6388643afb90a307c6bf2c"
Sep 17 23:15:50.074385 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:50.074367    2566 generic.go:356] "Generic (PLEG): container finished" podID="6591d17a-2227-4fc6-bcc5-cdc82be2605e" containerID="a2fda411c90f11200d67127a09f39da3fda5e51ecb400d07b22e1e884cc7ef47" exitCode=0
Sep 17 23:15:50.074432 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:50.074400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq" event={"ID":"6591d17a-2227-4fc6-bcc5-cdc82be2605e","Type":"ContainerDied","Data":"a2fda411c90f11200d67127a09f39da3fda5e51ecb400d07b22e1e884cc7ef47"}
Sep 17 23:15:50.074676 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:50.074660    2566 scope.go:117] "RemoveContainer" containerID="a2fda411c90f11200d67127a09f39da3fda5e51ecb400d07b22e1e884cc7ef47"
Sep 17 23:15:51.080656 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:51.080616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xwkcq" event={"ID":"6591d17a-2227-4fc6-bcc5-cdc82be2605e","Type":"ContainerStarted","Data":"c39d0174ffdf62c1362d5fed2bcbeac3887aac464e2f58d919d0e862dd2a9ad8"}
Sep 17 23:15:51.082728 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:15:51.082701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqk5" event={"ID":"ed0528cc-e40e-4841-bcd7-4cb9b0f5a50b","Type":"ContainerStarted","Data":"89e80eb759ea03cf16fe4370c1a4cd73d9d71f29d1bb2fac78e4f5387d9c98be"}
Sep 17 23:16:18.800524 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:18.800483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:18.818460 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:18.818432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:19.205472 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:19.205404    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:36.805824 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:36.805717    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:16:36.806474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:36.806212    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="prometheus" containerID="cri-o://e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2" gracePeriod=600
Sep 17 23:16:36.806474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:36.806211    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy" containerID="cri-o://3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468" gracePeriod=600
Sep 17 23:16:36.806474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:36.806257    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="config-reloader" containerID="cri-o://fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa" gracePeriod=600
Sep 17 23:16:36.806474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:36.806211    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-web" containerID="cri-o://d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570" gracePeriod=600
Sep 17 23:16:36.806474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:36.806211    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32" gracePeriod=600
Sep 17 23:16:36.806474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:36.806243    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="thanos-sidecar" containerID="cri-o://95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f" gracePeriod=600
Sep 17 23:16:37.049675 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.049652    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.105379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105308    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-tls-assets\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105345    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-metrics-client-ca\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-trusted-ca-bundle\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105529    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-2\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-config\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105610    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-grpc-tls\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105643    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9c2\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-kube-api-access-gr9c2\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-1\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105711    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-web-config\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105740    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-kubelet-serving-ca-bundle\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105771    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-kube-rbac-proxy\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.105783 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105810    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-tls\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105816    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-db\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105865    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-metrics-client-certs\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105896    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-serving-certs-ca-bundle\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105930    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-config-out\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.105982    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-thanos-prometheus-http-client-file\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.106029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-0\") pod \"ced819dd-27b0-4c70-b3f4-5992ff109965\" (UID: \"ced819dd-27b0-4c70-b3f4-5992ff109965\") "
Sep 17 23:16:37.106535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.106131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:37.106535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.106284    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-metrics-client-ca\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.106535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.106361    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.106535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.106377    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.107757 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.106951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:37.107757 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.107520    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:37.108394 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.108367    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:37.109743 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.109716    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:16:37.109850 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.109805    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-kube-api-access-gr9c2" (OuterVolumeSpecName: "kube-api-access-gr9c2") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "kube-api-access-gr9c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:37.110081 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.110055    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:37.110182 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.110159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-config" (OuterVolumeSpecName: "config") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.110302 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.110253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:37.110357 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.110325    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.110421 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.110394    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-config-out" (OuterVolumeSpecName: "config-out") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:16:37.110656 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.110618    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.110749 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.110667    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.111334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.111309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.111804 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.111779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.111926 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.111908    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.112391 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.112368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.120137 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.120119    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-web-config" (OuterVolumeSpecName: "web-config") pod "ced819dd-27b0-4c70-b3f4-5992ff109965" (UID: "ced819dd-27b0-4c70-b3f4-5992ff109965"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:37.207498 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207477    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207498 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207498    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-config-out\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207509    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207520    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207529    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-tls-assets\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207537    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207546    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-config\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207553    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-grpc-tls\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207562    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9c2\" (UniqueName: \"kubernetes.io/projected/ced819dd-27b0-4c70-b3f4-5992ff109965-kube-api-access-gr9c2\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207570    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207593    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-web-config\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207601    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207610    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-kube-rbac-proxy\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207619    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207627    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ced819dd-27b0-4c70-b3f4-5992ff109965-prometheus-k8s-db\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207635    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ced819dd-27b0-4c70-b3f4-5992ff109965-secret-metrics-client-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.207643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.207643    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced819dd-27b0-4c70-b3f4-5992ff109965-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:16:37.266039 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266010    2566 generic.go:356] "Generic (PLEG): container finished" podID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32" exitCode=0
Sep 17 23:16:37.266039 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266034    2566 generic.go:356] "Generic (PLEG): container finished" podID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468" exitCode=0
Sep 17 23:16:37.266167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266043    2566 generic.go:356] "Generic (PLEG): container finished" podID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570" exitCode=0
Sep 17 23:16:37.266167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266050    2566 generic.go:356] "Generic (PLEG): container finished" podID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f" exitCode=0
Sep 17 23:16:37.266167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266057    2566 generic.go:356] "Generic (PLEG): container finished" podID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa" exitCode=0
Sep 17 23:16:37.266167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266063    2566 generic.go:356] "Generic (PLEG): container finished" podID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2" exitCode=0
Sep 17 23:16:37.266167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"}
Sep 17 23:16:37.266316 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266091    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.266316 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266180    2566 scope.go:117] "RemoveContainer" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.266316 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"}
Sep 17 23:16:37.266316 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"}
Sep 17 23:16:37.266469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"}
Sep 17 23:16:37.266469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"}
Sep 17 23:16:37.266469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"}
Sep 17 23:16:37.266568 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.266467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ced819dd-27b0-4c70-b3f4-5992ff109965","Type":"ContainerDied","Data":"a3f4c29ec4e3d5f71d64baa22d1f3953942f7946c466667627eea6263fced109"}
Sep 17 23:16:37.273629 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.273611    2566 scope.go:117] "RemoveContainer" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.281056 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.281039    2566 scope.go:117] "RemoveContainer" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.287088 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.287076    2566 scope.go:117] "RemoveContainer" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.291132 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.291110    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:16:37.294049 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.294032    2566 scope.go:117] "RemoveContainer" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.295942 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.295918    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:16:37.302400 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.302378    2566 scope.go:117] "RemoveContainer" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.309435 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.309420    2566 scope.go:117] "RemoveContainer" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.316157 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.316141    2566 scope.go:117] "RemoveContainer" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.316421 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:16:37.316404    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": container with ID starting with d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32 not found: ID does not exist" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.316472 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.316436    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"} err="failed to get container status \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": rpc error: code = NotFound desc = could not find container \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": container with ID starting with d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32 not found: ID does not exist"
Sep 17 23:16:37.316472 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.316452    2566 scope.go:117] "RemoveContainer" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.316714 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:16:37.316697    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": container with ID starting with 3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468 not found: ID does not exist" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.316769 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.316718    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"} err="failed to get container status \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": rpc error: code = NotFound desc = could not find container \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": container with ID starting with 3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468 not found: ID does not exist"
Sep 17 23:16:37.316769 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.316734    2566 scope.go:117] "RemoveContainer" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.316969 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:16:37.316951    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": container with ID starting with d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570 not found: ID does not exist" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.317007 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.316973    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"} err="failed to get container status \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": rpc error: code = NotFound desc = could not find container \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": container with ID starting with d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570 not found: ID does not exist"
Sep 17 23:16:37.317007 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.316987    2566 scope.go:117] "RemoveContainer" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.317202 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:16:37.317186    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": container with ID starting with 95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f not found: ID does not exist" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.317250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.317204    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"} err="failed to get container status \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": rpc error: code = NotFound desc = could not find container \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": container with ID starting with 95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f not found: ID does not exist"
Sep 17 23:16:37.317250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.317216    2566 scope.go:117] "RemoveContainer" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.317425 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:16:37.317409    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": container with ID starting with fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa not found: ID does not exist" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.317459 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.317428    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"} err="failed to get container status \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": rpc error: code = NotFound desc = could not find container \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": container with ID starting with fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa not found: ID does not exist"
Sep 17 23:16:37.317459 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.317441    2566 scope.go:117] "RemoveContainer" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.317890 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:16:37.317848    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": container with ID starting with e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2 not found: ID does not exist" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.317890 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.317877    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"} err="failed to get container status \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": rpc error: code = NotFound desc = could not find container \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": container with ID starting with e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2 not found: ID does not exist"
Sep 17 23:16:37.318040 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.317901    2566 scope.go:117] "RemoveContainer" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.318330 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:16:37.318307    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": container with ID starting with 4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1 not found: ID does not exist" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.318442 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.318335    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"} err="failed to get container status \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": rpc error: code = NotFound desc = could not find container \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": container with ID starting with 4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1 not found: ID does not exist"
Sep 17 23:16:37.318442 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.318356    2566 scope.go:117] "RemoveContainer" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.318769 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.318750    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"} err="failed to get container status \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": rpc error: code = NotFound desc = could not find container \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": container with ID starting with d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32 not found: ID does not exist"
Sep 17 23:16:37.318832 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.318770    2566 scope.go:117] "RemoveContainer" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.319017 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.318999    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"} err="failed to get container status \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": rpc error: code = NotFound desc = could not find container \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": container with ID starting with 3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468 not found: ID does not exist"
Sep 17 23:16:37.319017 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319016    2566 scope.go:117] "RemoveContainer" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.319282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319263    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"} err="failed to get container status \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": rpc error: code = NotFound desc = could not find container \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": container with ID starting with d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570 not found: ID does not exist"
Sep 17 23:16:37.319282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319282    2566 scope.go:117] "RemoveContainer" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.319534 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319515    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"} err="failed to get container status \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": rpc error: code = NotFound desc = could not find container \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": container with ID starting with 95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f not found: ID does not exist"
Sep 17 23:16:37.319534 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319534    2566 scope.go:117] "RemoveContainer" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.319775 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319759    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"} err="failed to get container status \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": rpc error: code = NotFound desc = could not find container \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": container with ID starting with fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa not found: ID does not exist"
Sep 17 23:16:37.319775 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319774    2566 scope.go:117] "RemoveContainer" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.320017 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.319999    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"} err="failed to get container status \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": rpc error: code = NotFound desc = could not find container \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": container with ID starting with e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2 not found: ID does not exist"
Sep 17 23:16:37.320073 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320017    2566 scope.go:117] "RemoveContainer" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.320114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320092    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:16:37.320246 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320226    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"} err="failed to get container status \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": rpc error: code = NotFound desc = could not find container \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": container with ID starting with 4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1 not found: ID does not exist"
Sep 17 23:16:37.320287 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320247    2566 scope.go:117] "RemoveContainer" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.320352 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320335    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="prometheus"
Sep 17 23:16:37.320352 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320350    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="prometheus"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320358    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="thanos-sidecar"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320364    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="thanos-sidecar"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320374    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="config-reloader"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320382    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="config-reloader"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320393    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-thanos"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320399    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-thanos"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320406    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-web"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320412    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-web"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320418    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="init-config-reloader"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320423    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="init-config-reloader"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320434    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320440    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320450    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" containerName="registry"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320457    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" containerName="registry"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320466    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"} err="failed to get container status \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": rpc error: code = NotFound desc = could not find container \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": container with ID starting with d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32 not found: ID does not exist"
Sep 17 23:16:37.320477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320483    2566 scope.go:117] "RemoveContainer" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320501    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="prometheus"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320508    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="thanos-sidecar"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320514    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b37b0f-b030-45ed-b87b-bf9f1d7a5b92" containerName="registry"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320520    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-web"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320528    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="config-reloader"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320535    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy-thanos"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320542    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" containerName="kube-rbac-proxy"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320750    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"} err="failed to get container status \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": rpc error: code = NotFound desc = could not find container \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": container with ID starting with 3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468 not found: ID does not exist"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320765    2566 scope.go:117] "RemoveContainer" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320965    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"} err="failed to get container status \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": rpc error: code = NotFound desc = could not find container \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": container with ID starting with d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570 not found: ID does not exist"
Sep 17 23:16:37.321004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.320977    2566 scope.go:117] "RemoveContainer" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.321417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321180    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"} err="failed to get container status \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": rpc error: code = NotFound desc = could not find container \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": container with ID starting with 95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f not found: ID does not exist"
Sep 17 23:16:37.321417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321201    2566 scope.go:117] "RemoveContainer" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.321509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321409    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"} err="failed to get container status \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": rpc error: code = NotFound desc = could not find container \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": container with ID starting with fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa not found: ID does not exist"
Sep 17 23:16:37.321509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321430    2566 scope.go:117] "RemoveContainer" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.321741 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321708    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"} err="failed to get container status \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": rpc error: code = NotFound desc = could not find container \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": container with ID starting with e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2 not found: ID does not exist"
Sep 17 23:16:37.321806 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321743    2566 scope.go:117] "RemoveContainer" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.321996 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321978    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"} err="failed to get container status \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": rpc error: code = NotFound desc = could not find container \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": container with ID starting with 4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1 not found: ID does not exist"
Sep 17 23:16:37.321996 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.321995    2566 scope.go:117] "RemoveContainer" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.322236 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.322219    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"} err="failed to get container status \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": rpc error: code = NotFound desc = could not find container \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": container with ID starting with d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32 not found: ID does not exist"
Sep 17 23:16:37.322280 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.322238    2566 scope.go:117] "RemoveContainer" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.322510 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.322487    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"} err="failed to get container status \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": rpc error: code = NotFound desc = could not find container \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": container with ID starting with 3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468 not found: ID does not exist"
Sep 17 23:16:37.322564 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.322511    2566 scope.go:117] "RemoveContainer" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.322870 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.322844    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"} err="failed to get container status \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": rpc error: code = NotFound desc = could not find container \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": container with ID starting with d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570 not found: ID does not exist"
Sep 17 23:16:37.322870 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.322871    2566 scope.go:117] "RemoveContainer" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.323139 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.323118    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"} err="failed to get container status \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": rpc error: code = NotFound desc = could not find container \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": container with ID starting with 95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f not found: ID does not exist"
Sep 17 23:16:37.323139 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.323140    2566 scope.go:117] "RemoveContainer" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.323379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.323362    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"} err="failed to get container status \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": rpc error: code = NotFound desc = could not find container \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": container with ID starting with fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa not found: ID does not exist"
Sep 17 23:16:37.323379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.323379    2566 scope.go:117] "RemoveContainer" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.323639 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.323613    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"} err="failed to get container status \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": rpc error: code = NotFound desc = could not find container \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": container with ID starting with e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2 not found: ID does not exist"
Sep 17 23:16:37.323739 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.323639    2566 scope.go:117] "RemoveContainer" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.324000 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.323976    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"} err="failed to get container status \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": rpc error: code = NotFound desc = could not find container \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": container with ID starting with 4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1 not found: ID does not exist"
Sep 17 23:16:37.324083 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324001    2566 scope.go:117] "RemoveContainer" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.324290 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324258    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"} err="failed to get container status \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": rpc error: code = NotFound desc = could not find container \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": container with ID starting with d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32 not found: ID does not exist"
Sep 17 23:16:37.324290 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324284    2566 scope.go:117] "RemoveContainer" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.324569 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324537    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"} err="failed to get container status \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": rpc error: code = NotFound desc = could not find container \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": container with ID starting with 3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468 not found: ID does not exist"
Sep 17 23:16:37.324569 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324557    2566 scope.go:117] "RemoveContainer" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.324733 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.324877 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324848    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"} err="failed to get container status \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": rpc error: code = NotFound desc = could not find container \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": container with ID starting with d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570 not found: ID does not exist"
Sep 17 23:16:37.324877 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.324872    2566 scope.go:117] "RemoveContainer" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.325164 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.325138    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"} err="failed to get container status \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": rpc error: code = NotFound desc = could not find container \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": container with ID starting with 95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f not found: ID does not exist"
Sep 17 23:16:37.325248 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.325165    2566 scope.go:117] "RemoveContainer" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.325458 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.325432    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"} err="failed to get container status \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": rpc error: code = NotFound desc = could not find container \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": container with ID starting with fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa not found: ID does not exist"
Sep 17 23:16:37.325525 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.325459    2566 scope.go:117] "RemoveContainer" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.325779 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.325755    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"} err="failed to get container status \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": rpc error: code = NotFound desc = could not find container \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": container with ID starting with e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2 not found: ID does not exist"
Sep 17 23:16:37.325844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.325782    2566 scope.go:117] "RemoveContainer" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.326054 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326034    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"} err="failed to get container status \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": rpc error: code = NotFound desc = could not find container \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": container with ID starting with 4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1 not found: ID does not exist"
Sep 17 23:16:37.326054 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326052    2566 scope.go:117] "RemoveContainer" containerID="d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"
Sep 17 23:16:37.326355 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326329    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32"} err="failed to get container status \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": rpc error: code = NotFound desc = could not find container \"d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32\": container with ID starting with d9b30746a92db4f200521040068fa2f314e874d157cbe40aaa38b0720f111e32 not found: ID does not exist"
Sep 17 23:16:37.326430 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326358    2566 scope.go:117] "RemoveContainer" containerID="3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"
Sep 17 23:16:37.326669 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326645    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468"} err="failed to get container status \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": rpc error: code = NotFound desc = could not find container \"3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468\": container with ID starting with 3df9e395c5c44323081cf4024ce87566c4137372347ce0baf55c7ab4f6b7f468 not found: ID does not exist"
Sep 17 23:16:37.326669 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326668    2566 scope.go:117] "RemoveContainer" containerID="d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"
Sep 17 23:16:37.326957 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326934    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570"} err="failed to get container status \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": rpc error: code = NotFound desc = could not find container \"d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570\": container with ID starting with d77f52726d7affdff3e4193f7418186f6fa4e9d23b8c53a65a5f35474dcbe570 not found: ID does not exist"
Sep 17 23:16:37.327036 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.326959    2566 scope.go:117] "RemoveContainer" containerID="95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"
Sep 17 23:16:37.327273 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.327256    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f"} err="failed to get container status \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": rpc error: code = NotFound desc = could not find container \"95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f\": container with ID starting with 95be381c7ca547cea78e05529605bc20a4dfb449c2c1887e5ebf0d878f36b05f not found: ID does not exist"
Sep 17 23:16:37.327273 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.327272    2566 scope.go:117] "RemoveContainer" containerID="fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"
Sep 17 23:16:37.327522 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.327298    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:16:37.327696 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.327547    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa"} err="failed to get container status \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": rpc error: code = NotFound desc = could not find container \"fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa\": container with ID starting with fa2dded3517b1c2e000ab0fd50df0123631e9330fd16eaeaff451c43d84b6faa not found: ID does not exist"
Sep 17 23:16:37.327696 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.327565    2566 scope.go:117] "RemoveContainer" containerID="e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"
Sep 17 23:16:37.327897 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.327874    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2"} err="failed to get container status \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": rpc error: code = NotFound desc = could not find container \"e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2\": container with ID starting with e511079663ebafaf187ce6114e57cb0bf8d92f7b8674dbdee4b7528a1e7c92e2 not found: ID does not exist"
Sep 17 23:16:37.327897 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.327899    2566 scope.go:117] "RemoveContainer" containerID="4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"
Sep 17 23:16:37.328331 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328175    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1"} err="failed to get container status \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": rpc error: code = NotFound desc = could not find container \"4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1\": container with ID starting with 4ddef6e731997781255526ffd73f36b61a48d713febd8179f2d92ab22c30bbf1 not found: ID does not exist"
Sep 17 23:16:37.328569 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:16:37.328569 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328515    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:16:37.328569 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328620    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:16:37.328569 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:16:37.329334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328750    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:16:37.329334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328829    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:16:37.329334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:16:37.329334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.328989    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:16:37.329334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.329004    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:16:37.329334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.329269    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:16:37.329334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.329271    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2d92w\""
Sep 17 23:16:37.329715 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.329386    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4dall3dgth676\""
Sep 17 23:16:37.329715 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.329551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:16:37.330024 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.330007    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:16:37.332063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.332040    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:16:37.333964 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.333945    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:16:37.342446 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.342427    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:16:37.409321 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-config\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409321 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp67c\" (UniqueName: \"kubernetes.io/projected/1d983559-4241-422c-89b9-0dac7d6b6870-kube-api-access-cp67c\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409545 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409545 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409545 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d983559-4241-422c-89b9-0dac7d6b6870-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409994 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409994 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d983559-4241-422c-89b9-0dac7d6b6870-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.409994 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.409898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510621 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d983559-4241-422c-89b9-0dac7d6b6870-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.510935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.510800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-config\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511375 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511353    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511529 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511529 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511529 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp67c\" (UniqueName: \"kubernetes.io/projected/1d983559-4241-422c-89b9-0dac7d6b6870-kube-api-access-cp67c\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511712 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511712 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511712 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511712 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511914 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d983559-4241-422c-89b9-0dac7d6b6870-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511914 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511914 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.511914 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.511805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.513334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.512980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.513432 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.513355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.513493 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.513432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.514084 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.514032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d983559-4241-422c-89b9-0dac7d6b6870-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.514190 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.514129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-config\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.515218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.514432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.515218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.514673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.515218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.514705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.515218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.514928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.515218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.515132    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.515218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.515166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.515736 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.515714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.516075 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.516055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.516312 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.516291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.516562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.516543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d983559-4241-422c-89b9-0dac7d6b6870-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.516651 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.516601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d983559-4241-422c-89b9-0dac7d6b6870-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.516651 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.516607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d983559-4241-422c-89b9-0dac7d6b6870-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.523750 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.523725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp67c\" (UniqueName: \"kubernetes.io/projected/1d983559-4241-422c-89b9-0dac7d6b6870-kube-api-access-cp67c\") pod \"prometheus-k8s-0\" (UID: \"1d983559-4241-422c-89b9-0dac7d6b6870\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.635834 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.635810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:16:37.910662 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:37.910635    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:16:37.912419 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:16:37.912386    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d983559_4241_422c_89b9_0dac7d6b6870.slice/crio-db8250e6c5305a9d8abfa2f1458a8425730a724c718d58a1316c2311f7a9b54e WatchSource:0}: Error finding container db8250e6c5305a9d8abfa2f1458a8425730a724c718d58a1316c2311f7a9b54e: Status 404 returned error can't find the container with id db8250e6c5305a9d8abfa2f1458a8425730a724c718d58a1316c2311f7a9b54e
Sep 17 23:16:38.012850 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:38.012826    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced819dd-27b0-4c70-b3f4-5992ff109965" path="/var/lib/kubelet/pods/ced819dd-27b0-4c70-b3f4-5992ff109965/volumes"
Sep 17 23:16:38.271608 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:38.271557    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d983559-4241-422c-89b9-0dac7d6b6870" containerID="cd28c6541d25aa68ce1eeb0bf14e8f9bf093f728f140a334e04d11b7f7a9be61" exitCode=0
Sep 17 23:16:38.271732 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:38.271643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerDied","Data":"cd28c6541d25aa68ce1eeb0bf14e8f9bf093f728f140a334e04d11b7f7a9be61"}
Sep 17 23:16:38.271732 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:38.271716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerStarted","Data":"db8250e6c5305a9d8abfa2f1458a8425730a724c718d58a1316c2311f7a9b54e"}
Sep 17 23:16:39.280237 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:39.280205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerStarted","Data":"2a487d166b8940a58ca8b9ce3fb7b36fdfbc84980f600421612516b398b0ff0b"}
Sep 17 23:16:39.280693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:39.280266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerStarted","Data":"3edf1a6556613ca4bbac4744006bbffcf807ce42c52f8b88e2e2ff3d66a9cf31"}
Sep 17 23:16:39.280693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:39.280298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerStarted","Data":"cbaf2a5106357826d56f7cee0a4dd1247a34609ddc8a0b403ca2f97ac9560693"}
Sep 17 23:16:39.280693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:39.280323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerStarted","Data":"e3c90286818f89f32cd298d82faa48b46d0b7165dbfe04077d0657782303929c"}
Sep 17 23:16:39.280693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:39.280346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerStarted","Data":"08343ba8779939b758869ff7b03412d98f7164cb0d654adbaeb9813160ab556a"}
Sep 17 23:16:39.280693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:39.280368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d983559-4241-422c-89b9-0dac7d6b6870","Type":"ContainerStarted","Data":"1bd322f17da3b3a297535b9768ed0ab3fdcab77090145b2a0afa5bc2a4b0aa8f"}
Sep 17 23:16:39.311248 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:39.311201    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.289545469 podStartE2EDuration="2.311186352s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.640883ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:39.308921425 +0000 UTC m=+359.760752925" watchObservedRunningTime="2026-09-17 23:16:39.311186352 +0000 UTC m=+359.763017840"
Sep 17 23:16:42.636964 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:16:42.636918    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:17:37.636916 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:17:37.636876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:17:37.652745 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:17:37.652717    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:17:38.523109 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:17:38.523082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:19:17.531459 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.531419    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"]
Sep 17 23:19:17.534718 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.534694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:17.537450 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.537427    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x67v5\""
Sep 17 23:19:17.537551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.537478    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 23:19:17.537617 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.537546    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 23:19:17.545363 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.545340    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"]
Sep 17 23:19:17.630303 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.630276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczv9\" (UniqueName: \"kubernetes.io/projected/75df9b6a-755a-49b5-a36f-13fa2d227bf2-kube-api-access-gczv9\") pod \"cert-manager-cainjector-559cdd8c87-jr4cf\" (UID: \"75df9b6a-755a-49b5-a36f-13fa2d227bf2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:17.630399 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.630344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75df9b6a-755a-49b5-a36f-13fa2d227bf2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jr4cf\" (UID: \"75df9b6a-755a-49b5-a36f-13fa2d227bf2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:17.731295 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.731269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75df9b6a-755a-49b5-a36f-13fa2d227bf2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jr4cf\" (UID: \"75df9b6a-755a-49b5-a36f-13fa2d227bf2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:17.731398 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.731304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczv9\" (UniqueName: \"kubernetes.io/projected/75df9b6a-755a-49b5-a36f-13fa2d227bf2-kube-api-access-gczv9\") pod \"cert-manager-cainjector-559cdd8c87-jr4cf\" (UID: \"75df9b6a-755a-49b5-a36f-13fa2d227bf2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:17.740420 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.740391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczv9\" (UniqueName: \"kubernetes.io/projected/75df9b6a-755a-49b5-a36f-13fa2d227bf2-kube-api-access-gczv9\") pod \"cert-manager-cainjector-559cdd8c87-jr4cf\" (UID: \"75df9b6a-755a-49b5-a36f-13fa2d227bf2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:17.741271 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.741242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75df9b6a-755a-49b5-a36f-13fa2d227bf2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jr4cf\" (UID: \"75df9b6a-755a-49b5-a36f-13fa2d227bf2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:17.845113 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:17.845060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"
Sep 17 23:19:18.113067 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:18.112987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf"]
Sep 17 23:19:18.115821 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:19:18.115794    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75df9b6a_755a_49b5_a36f_13fa2d227bf2.slice/crio-a1cc0f646e80ca80b768537d0c450c3ef6221fb106ddb9ef6ac628d7729bb54b WatchSource:0}: Error finding container a1cc0f646e80ca80b768537d0c450c3ef6221fb106ddb9ef6ac628d7729bb54b: Status 404 returned error can't find the container with id a1cc0f646e80ca80b768537d0c450c3ef6221fb106ddb9ef6ac628d7729bb54b
Sep 17 23:19:18.117856 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:18.117834    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:19:18.915102 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:18.915042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf" event={"ID":"75df9b6a-755a-49b5-a36f-13fa2d227bf2","Type":"ContainerStarted","Data":"a1cc0f646e80ca80b768537d0c450c3ef6221fb106ddb9ef6ac628d7729bb54b"}
Sep 17 23:19:21.929472 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:21.929428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf" event={"ID":"75df9b6a-755a-49b5-a36f-13fa2d227bf2","Type":"ContainerStarted","Data":"0bac4e852bd43c01be15b75a0a1695cfbe9d407cacabdba37152f91511f962ac"}
Sep 17 23:19:21.948364 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:21.948309    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr4cf" podStartSLOduration=2.077904437 podStartE2EDuration="4.948290667s" totalImagesPullingTime="2.87038623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:21.946880885 +0000 UTC m=+522.398712374" watchObservedRunningTime="2026-09-17 23:19:21.948290667 +0000 UTC m=+522.400122155"
Sep 17 23:19:41.880817 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.880702    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"]
Sep 17 23:19:41.884492 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.884464    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:41.888331 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.888305    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-s9tgh\""
Sep 17 23:19:41.888493 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.888472    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:19:41.888622 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.888360    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 23:19:41.894289 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.894260    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"]
Sep 17 23:19:41.898770 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.898732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e153cda8-4e06-4e07-a0ba-39a562590ab9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jgdk4\" (UID: \"e153cda8-4e06-4e07-a0ba-39a562590ab9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:41.898935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.898795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kft\" (UniqueName: \"kubernetes.io/projected/e153cda8-4e06-4e07-a0ba-39a562590ab9-kube-api-access-g2kft\") pod \"openshift-lws-operator-fd8ccff4c-jgdk4\" (UID: \"e153cda8-4e06-4e07-a0ba-39a562590ab9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:41.999530 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.999499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e153cda8-4e06-4e07-a0ba-39a562590ab9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jgdk4\" (UID: \"e153cda8-4e06-4e07-a0ba-39a562590ab9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:41.999696 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.999546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kft\" (UniqueName: \"kubernetes.io/projected/e153cda8-4e06-4e07-a0ba-39a562590ab9-kube-api-access-g2kft\") pod \"openshift-lws-operator-fd8ccff4c-jgdk4\" (UID: \"e153cda8-4e06-4e07-a0ba-39a562590ab9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:41.999891 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:41.999873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e153cda8-4e06-4e07-a0ba-39a562590ab9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jgdk4\" (UID: \"e153cda8-4e06-4e07-a0ba-39a562590ab9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:42.007720 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:42.007696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kft\" (UniqueName: \"kubernetes.io/projected/e153cda8-4e06-4e07-a0ba-39a562590ab9-kube-api-access-g2kft\") pod \"openshift-lws-operator-fd8ccff4c-jgdk4\" (UID: \"e153cda8-4e06-4e07-a0ba-39a562590ab9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:42.199983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:42.199912    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"
Sep 17 23:19:42.467121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:42.467040    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4"]
Sep 17 23:19:42.470602 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:19:42.470557    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode153cda8_4e06_4e07_a0ba_39a562590ab9.slice/crio-2fabd15372fa5e1fb0f795895a7f187bed0d80f5b661a68806b1b27c4f8c6d0b WatchSource:0}: Error finding container 2fabd15372fa5e1fb0f795895a7f187bed0d80f5b661a68806b1b27c4f8c6d0b: Status 404 returned error can't find the container with id 2fabd15372fa5e1fb0f795895a7f187bed0d80f5b661a68806b1b27c4f8c6d0b
Sep 17 23:19:43.021608 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:43.021550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4" event={"ID":"e153cda8-4e06-4e07-a0ba-39a562590ab9","Type":"ContainerStarted","Data":"2fabd15372fa5e1fb0f795895a7f187bed0d80f5b661a68806b1b27c4f8c6d0b"}
Sep 17 23:19:45.033306 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:45.033266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4" event={"ID":"e153cda8-4e06-4e07-a0ba-39a562590ab9","Type":"ContainerStarted","Data":"2a788803540a99fd306bd32c3ca789f8735a3ec0a6222ceafd9ef03e830b0387"}
Sep 17 23:19:45.052282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:45.052230    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jgdk4" podStartSLOduration=1.6013455890000001 podStartE2EDuration="4.052210551s" totalImagesPullingTime="2.450864962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:45.049741268 +0000 UTC m=+545.501572755" watchObservedRunningTime="2026-09-17 23:19:45.052210551 +0000 UTC m=+545.504042042"
Sep 17 23:19:56.714591 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.714536    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"]
Sep 17 23:19:56.718860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.718838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.721561 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.721540    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 23:19:56.722728 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.722699    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 23:19:56.722826 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.722702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 23:19:56.722907 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.722888    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cww64\""
Sep 17 23:19:56.727190 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.727165    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"]
Sep 17 23:19:56.808478 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.808451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46594425-5c87-4915-a01f-c58519b3fbd2-cert\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.808627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.808493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/46594425-5c87-4915-a01f-c58519b3fbd2-manager-config\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.808627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.808517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hjz\" (UniqueName: \"kubernetes.io/projected/46594425-5c87-4915-a01f-c58519b3fbd2-kube-api-access-k6hjz\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.808627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.808614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46594425-5c87-4915-a01f-c58519b3fbd2-metrics-cert\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.909653 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.909622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46594425-5c87-4915-a01f-c58519b3fbd2-cert\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.909822 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.909688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/46594425-5c87-4915-a01f-c58519b3fbd2-manager-config\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.909822 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.909730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hjz\" (UniqueName: \"kubernetes.io/projected/46594425-5c87-4915-a01f-c58519b3fbd2-kube-api-access-k6hjz\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.909822 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.909782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46594425-5c87-4915-a01f-c58519b3fbd2-metrics-cert\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.911055 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.911028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/46594425-5c87-4915-a01f-c58519b3fbd2-manager-config\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.912650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.912626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46594425-5c87-4915-a01f-c58519b3fbd2-cert\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.912800 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.912781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46594425-5c87-4915-a01f-c58519b3fbd2-metrics-cert\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:56.920103 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:56.920074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hjz\" (UniqueName: \"kubernetes.io/projected/46594425-5c87-4915-a01f-c58519b3fbd2-kube-api-access-k6hjz\") pod \"lws-controller-manager-6944fcc464-565sl\" (UID: \"46594425-5c87-4915-a01f-c58519b3fbd2\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:57.027816 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:57.027786    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:57.339825 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:57.339799    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"]
Sep 17 23:19:57.342611 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:19:57.342568    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46594425_5c87_4915_a01f_c58519b3fbd2.slice/crio-180bbd2ecf7ef8c757ee160b580c78af0b36f4b454da8232c44c8707e8e58a83 WatchSource:0}: Error finding container 180bbd2ecf7ef8c757ee160b580c78af0b36f4b454da8232c44c8707e8e58a83: Status 404 returned error can't find the container with id 180bbd2ecf7ef8c757ee160b580c78af0b36f4b454da8232c44c8707e8e58a83
Sep 17 23:19:58.106137 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:58.106095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl" event={"ID":"46594425-5c87-4915-a01f-c58519b3fbd2","Type":"ContainerStarted","Data":"180bbd2ecf7ef8c757ee160b580c78af0b36f4b454da8232c44c8707e8e58a83"}
Sep 17 23:19:59.111523 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:59.111443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl" event={"ID":"46594425-5c87-4915-a01f-c58519b3fbd2","Type":"ContainerStarted","Data":"63e4daa3a786ffc96e2f7d10e768d68e117c414d098ae05297d2dd98fd8886f3"}
Sep 17 23:19:59.111891 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:59.111525    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:19:59.128201 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:19:59.128137    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl" podStartSLOduration=1.70164475 podStartE2EDuration="3.128120296s" totalImagesPullingTime="1.426475546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:59.127002586 +0000 UTC m=+559.578834072" watchObservedRunningTime="2026-09-17 23:19:59.128120296 +0000 UTC m=+559.579951785"
Sep 17 23:20:10.117561 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:10.117531    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-565sl"
Sep 17 23:20:39.987968 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:39.987936    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:20:39.988419 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:39.988164    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:20:39.996653 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:39.996627    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:20:39.997061 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:39.997040    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:20:46.041365 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.041327    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"]
Sep 17 23:20:46.044728 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.044704    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"
Sep 17 23:20:46.047428 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.047398    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 23:20:46.047555 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.047434    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mt5lg\""
Sep 17 23:20:46.047555 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.047434    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 23:20:46.047555 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.047402    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 23:20:46.058451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.058429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"]
Sep 17 23:20:46.176699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.176666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jgq\" (UniqueName: \"kubernetes.io/projected/995f2cd8-7ab1-4ddd-a0ef-418d2f591ced-kube-api-access-m9jgq\") pod \"dns-operator-controller-manager-65dc8f76f5-4q6c9\" (UID: \"995f2cd8-7ab1-4ddd-a0ef-418d2f591ced\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"
Sep 17 23:20:46.277923 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.277898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jgq\" (UniqueName: \"kubernetes.io/projected/995f2cd8-7ab1-4ddd-a0ef-418d2f591ced-kube-api-access-m9jgq\") pod \"dns-operator-controller-manager-65dc8f76f5-4q6c9\" (UID: \"995f2cd8-7ab1-4ddd-a0ef-418d2f591ced\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"
Sep 17 23:20:46.298888 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.298820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jgq\" (UniqueName: \"kubernetes.io/projected/995f2cd8-7ab1-4ddd-a0ef-418d2f591ced-kube-api-access-m9jgq\") pod \"dns-operator-controller-manager-65dc8f76f5-4q6c9\" (UID: \"995f2cd8-7ab1-4ddd-a0ef-418d2f591ced\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"
Sep 17 23:20:46.355155 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.355123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"
Sep 17 23:20:46.628318 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:46.628298    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"]
Sep 17 23:20:46.630445 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:20:46.630413    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995f2cd8_7ab1_4ddd_a0ef_418d2f591ced.slice/crio-8a6ff2aa50ca60148217e6aeeb73cbd3f1420940fdc6b40068187804883c452f WatchSource:0}: Error finding container 8a6ff2aa50ca60148217e6aeeb73cbd3f1420940fdc6b40068187804883c452f: Status 404 returned error can't find the container with id 8a6ff2aa50ca60148217e6aeeb73cbd3f1420940fdc6b40068187804883c452f
Sep 17 23:20:47.303530 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:47.303489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9" event={"ID":"995f2cd8-7ab1-4ddd-a0ef-418d2f591ced","Type":"ContainerStarted","Data":"8a6ff2aa50ca60148217e6aeeb73cbd3f1420940fdc6b40068187804883c452f"}
Sep 17 23:20:50.318483 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:50.318445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9" event={"ID":"995f2cd8-7ab1-4ddd-a0ef-418d2f591ced","Type":"ContainerStarted","Data":"dfa651a58711b64cc9341ad06922976aeef339b8892a09b8a23dae624a536214"}
Sep 17 23:20:50.320821 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:50.318524    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"
Sep 17 23:20:50.341378 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:50.341334    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9" podStartSLOduration=1.6613605900000001 podStartE2EDuration="4.341318735s" totalImagesPullingTime="2.679958145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:50.340235213 +0000 UTC m=+610.792066700" watchObservedRunningTime="2026-09-17 23:20:50.341318735 +0000 UTC m=+610.793150221"
Sep 17 23:20:52.701567 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.701531    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"]
Sep 17 23:20:52.704669 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.704653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:52.707262 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.707238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4nfj7\""
Sep 17 23:20:52.715173 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.715151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"]
Sep 17 23:20:52.828860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.828832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8js\" (UniqueName: \"kubernetes.io/projected/532fa764-0bdc-43cd-9015-98e2da6879f4-kube-api-access-lf8js\") pod \"kuadrant-operator-controller-manager-6d4cb79c-qjk4h\" (UID: \"532fa764-0bdc-43cd-9015-98e2da6879f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:52.828963 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.828871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/532fa764-0bdc-43cd-9015-98e2da6879f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-qjk4h\" (UID: \"532fa764-0bdc-43cd-9015-98e2da6879f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:52.929438 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.929410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8js\" (UniqueName: \"kubernetes.io/projected/532fa764-0bdc-43cd-9015-98e2da6879f4-kube-api-access-lf8js\") pod \"kuadrant-operator-controller-manager-6d4cb79c-qjk4h\" (UID: \"532fa764-0bdc-43cd-9015-98e2da6879f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:52.929531 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.929447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/532fa764-0bdc-43cd-9015-98e2da6879f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-qjk4h\" (UID: \"532fa764-0bdc-43cd-9015-98e2da6879f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:52.929814 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.929796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/532fa764-0bdc-43cd-9015-98e2da6879f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-qjk4h\" (UID: \"532fa764-0bdc-43cd-9015-98e2da6879f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:52.937556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:52.937536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8js\" (UniqueName: \"kubernetes.io/projected/532fa764-0bdc-43cd-9015-98e2da6879f4-kube-api-access-lf8js\") pod \"kuadrant-operator-controller-manager-6d4cb79c-qjk4h\" (UID: \"532fa764-0bdc-43cd-9015-98e2da6879f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:53.016269 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:53.016244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:20:53.317107 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:53.317082    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"]
Sep 17 23:20:53.319767 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:20:53.319740    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532fa764_0bdc_43cd_9015_98e2da6879f4.slice/crio-363e260f9fbbcc5d6b6f02f3db75a9dec7e05047e78c77a44061d41442cbf3f4 WatchSource:0}: Error finding container 363e260f9fbbcc5d6b6f02f3db75a9dec7e05047e78c77a44061d41442cbf3f4: Status 404 returned error can't find the container with id 363e260f9fbbcc5d6b6f02f3db75a9dec7e05047e78c77a44061d41442cbf3f4
Sep 17 23:20:53.332131 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:20:53.332105    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h" event={"ID":"532fa764-0bdc-43cd-9015-98e2da6879f4","Type":"ContainerStarted","Data":"363e260f9fbbcc5d6b6f02f3db75a9dec7e05047e78c77a44061d41442cbf3f4"}
Sep 17 23:21:00.364289 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:00.364201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h" event={"ID":"532fa764-0bdc-43cd-9015-98e2da6879f4","Type":"ContainerStarted","Data":"33bde889a9d82a04bd4b1cbd54465be0bde7eecae9c2536e4f38b9965e674f2d"}
Sep 17 23:21:00.364667 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:00.364311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:21:00.384768 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:00.384727    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h" podStartSLOduration=1.638857901 podStartE2EDuration="8.384715778s" totalImagesPullingTime="6.745857877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:00.384369223 +0000 UTC m=+620.836200714" watchObservedRunningTime="2026-09-17 23:21:00.384715778 +0000 UTC m=+620.836547266"
Sep 17 23:21:01.324814 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:01.324778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4q6c9"
Sep 17 23:21:11.371840 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:11.371760    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-qjk4h"
Sep 17 23:21:43.448484 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.448444    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gnw47"]
Sep 17 23:21:43.451214 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.451191    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47"
Sep 17 23:21:43.453646 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.453625    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hvkgj\""
Sep 17 23:21:43.457830 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.457799    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gnw47"]
Sep 17 23:21:43.492671 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.492643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6f4\" (UniqueName: \"kubernetes.io/projected/e5d193c9-6ca3-4d03-bbe1-d9c5927375e3-kube-api-access-cl6f4\") pod \"authorino-7bcfd8f9cd-gnw47\" (UID: \"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47"
Sep 17 23:21:43.593818 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.593792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6f4\" (UniqueName: \"kubernetes.io/projected/e5d193c9-6ca3-4d03-bbe1-d9c5927375e3-kube-api-access-cl6f4\") pod \"authorino-7bcfd8f9cd-gnw47\" (UID: \"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47"
Sep 17 23:21:43.607845 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.607822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6f4\" (UniqueName: \"kubernetes.io/projected/e5d193c9-6ca3-4d03-bbe1-d9c5927375e3-kube-api-access-cl6f4\") pod \"authorino-7bcfd8f9cd-gnw47\" (UID: \"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47"
Sep 17 23:21:43.761976 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:43.761949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47"
Sep 17 23:21:44.028954 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:44.028888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gnw47"]
Sep 17 23:21:44.032300 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:21:44.032274    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5d193c9_6ca3_4d03_bbe1_d9c5927375e3.slice/crio-5d82d2266ebf607846c588023f00d99848b0c4f0749ed86150575c6022aee0d3 WatchSource:0}: Error finding container 5d82d2266ebf607846c588023f00d99848b0c4f0749ed86150575c6022aee0d3: Status 404 returned error can't find the container with id 5d82d2266ebf607846c588023f00d99848b0c4f0749ed86150575c6022aee0d3
Sep 17 23:21:44.568796 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:44.568757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47" event={"ID":"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3","Type":"ContainerStarted","Data":"5d82d2266ebf607846c588023f00d99848b0c4f0749ed86150575c6022aee0d3"}
Sep 17 23:21:46.581149 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:46.581100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47" event={"ID":"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3","Type":"ContainerStarted","Data":"8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342"}
Sep 17 23:21:46.596064 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:46.595994    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47" podStartSLOduration=2.009919076 podStartE2EDuration="3.595976958s" totalImagesPullingTime="1.586057882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:46.595003408 +0000 UTC m=+667.046834897" watchObservedRunningTime="2026-09-17 23:21:46.595976958 +0000 UTC m=+667.047808445"
Sep 17 23:21:47.375340 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:47.375310    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gnw47"]
Sep 17 23:21:48.591000 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:48.590936    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47" podUID="e5d193c9-6ca3-4d03-bbe1-d9c5927375e3" containerName="authorino" containerID="cri-o://8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342" gracePeriod=30
Sep 17 23:21:48.827313 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:48.827292    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47"
Sep 17 23:21:48.931951 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:48.931887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl6f4\" (UniqueName: \"kubernetes.io/projected/e5d193c9-6ca3-4d03-bbe1-d9c5927375e3-kube-api-access-cl6f4\") pod \"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3\" (UID: \"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3\") "
Sep 17 23:21:48.933951 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:48.933928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d193c9-6ca3-4d03-bbe1-d9c5927375e3-kube-api-access-cl6f4" (OuterVolumeSpecName: "kube-api-access-cl6f4") pod "e5d193c9-6ca3-4d03-bbe1-d9c5927375e3" (UID: "e5d193c9-6ca3-4d03-bbe1-d9c5927375e3"). InnerVolumeSpecName "kube-api-access-cl6f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:49.033200 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.033173    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl6f4\" (UniqueName: \"kubernetes.io/projected/e5d193c9-6ca3-4d03-bbe1-d9c5927375e3-kube-api-access-cl6f4\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:21:49.598670 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.598639    2566 generic.go:356] "Generic (PLEG): container finished" podID="e5d193c9-6ca3-4d03-bbe1-d9c5927375e3" containerID="8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342" exitCode=0
Sep 17 23:21:49.599026 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.598691    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47"
Sep 17 23:21:49.599026 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.598730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47" event={"ID":"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3","Type":"ContainerDied","Data":"8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342"}
Sep 17 23:21:49.599026 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.598780    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gnw47" event={"ID":"e5d193c9-6ca3-4d03-bbe1-d9c5927375e3","Type":"ContainerDied","Data":"5d82d2266ebf607846c588023f00d99848b0c4f0749ed86150575c6022aee0d3"}
Sep 17 23:21:49.599026 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.598804    2566 scope.go:117] "RemoveContainer" containerID="8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342"
Sep 17 23:21:49.608607 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.608559    2566 scope.go:117] "RemoveContainer" containerID="8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342"
Sep 17 23:21:49.608956 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:21:49.608932    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342\": container with ID starting with 8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342 not found: ID does not exist" containerID="8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342"
Sep 17 23:21:49.609021 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.608965    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342"} err="failed to get container status \"8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342\": rpc error: code = NotFound desc = could not find container \"8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342\": container with ID starting with 8c77f46cb407d43ae85e380cd31d10c5edd555b4bb9a4fe9d3edb7f98afeb342 not found: ID does not exist"
Sep 17 23:21:49.619450 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.619419    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gnw47"]
Sep 17 23:21:49.622536 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:49.622509    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gnw47"]
Sep 17 23:21:50.014087 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:21:50.014045    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d193c9-6ca3-4d03-bbe1-d9c5927375e3" path="/var/lib/kubelet/pods/e5d193c9-6ca3-4d03-bbe1-d9c5927375e3/volumes"
Sep 17 23:22:04.688588 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.688553    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-jrtvd"]
Sep 17 23:22:04.688940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.688873    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d193c9-6ca3-4d03-bbe1-d9c5927375e3" containerName="authorino"
Sep 17 23:22:04.688940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.688885    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d193c9-6ca3-4d03-bbe1-d9c5927375e3" containerName="authorino"
Sep 17 23:22:04.689014 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.688956    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d193c9-6ca3-4d03-bbe1-d9c5927375e3" containerName="authorino"
Sep 17 23:22:04.694014 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.693995    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:04.696785 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.696758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 23:22:04.697479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.697458    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hvkgj\""
Sep 17 23:22:04.698207 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.698185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-jrtvd"]
Sep 17 23:22:04.843848 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.843820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d99d8a01-0aaf-41ea-8d71-d51925c8d953-tls-cert\") pod \"authorino-6d9898bd99-jrtvd\" (UID: \"d99d8a01-0aaf-41ea-8d71-d51925c8d953\") " pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:04.843953 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.843878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvhc\" (UniqueName: \"kubernetes.io/projected/d99d8a01-0aaf-41ea-8d71-d51925c8d953-kube-api-access-mcvhc\") pod \"authorino-6d9898bd99-jrtvd\" (UID: \"d99d8a01-0aaf-41ea-8d71-d51925c8d953\") " pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:04.944619 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.944538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d99d8a01-0aaf-41ea-8d71-d51925c8d953-tls-cert\") pod \"authorino-6d9898bd99-jrtvd\" (UID: \"d99d8a01-0aaf-41ea-8d71-d51925c8d953\") " pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:04.944619 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.944601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvhc\" (UniqueName: \"kubernetes.io/projected/d99d8a01-0aaf-41ea-8d71-d51925c8d953-kube-api-access-mcvhc\") pod \"authorino-6d9898bd99-jrtvd\" (UID: \"d99d8a01-0aaf-41ea-8d71-d51925c8d953\") " pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:04.947077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.947052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d99d8a01-0aaf-41ea-8d71-d51925c8d953-tls-cert\") pod \"authorino-6d9898bd99-jrtvd\" (UID: \"d99d8a01-0aaf-41ea-8d71-d51925c8d953\") " pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:04.952585 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:04.952554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvhc\" (UniqueName: \"kubernetes.io/projected/d99d8a01-0aaf-41ea-8d71-d51925c8d953-kube-api-access-mcvhc\") pod \"authorino-6d9898bd99-jrtvd\" (UID: \"d99d8a01-0aaf-41ea-8d71-d51925c8d953\") " pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:05.004315 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:05.004294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-jrtvd"
Sep 17 23:22:05.274673 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:05.274595    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-jrtvd"]
Sep 17 23:22:05.277211 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:22:05.277181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd99d8a01_0aaf_41ea_8d71_d51925c8d953.slice/crio-e075cd16b4c5e17f192cf6522441391b8bf7d5c1dfeb57fde4837d638f5266b8 WatchSource:0}: Error finding container e075cd16b4c5e17f192cf6522441391b8bf7d5c1dfeb57fde4837d638f5266b8: Status 404 returned error can't find the container with id e075cd16b4c5e17f192cf6522441391b8bf7d5c1dfeb57fde4837d638f5266b8
Sep 17 23:22:05.682625 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:05.682545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-jrtvd" event={"ID":"d99d8a01-0aaf-41ea-8d71-d51925c8d953","Type":"ContainerStarted","Data":"e075cd16b4c5e17f192cf6522441391b8bf7d5c1dfeb57fde4837d638f5266b8"}
Sep 17 23:22:06.688671 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:06.688628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-jrtvd" event={"ID":"d99d8a01-0aaf-41ea-8d71-d51925c8d953","Type":"ContainerStarted","Data":"6e7d8df183763ba1f4b586e358a3abd80b14bdea9b797675147d96e60f1c0be1"}
Sep 17 23:22:06.704956 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:22:06.704913    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-jrtvd" podStartSLOduration=2.29889724 podStartE2EDuration="2.704898633s" totalImagesPullingTime="406.001393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:06.703787283 +0000 UTC m=+687.155618771" watchObservedRunningTime="2026-09-17 23:22:06.704898633 +0000 UTC m=+687.156730156"
Sep 17 23:23:50.871188 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.871153    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-hk6j7"]
Sep 17 23:23:50.874841 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.874813    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:50.877869 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.877838    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:23:50.878091 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.878064    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:23:50.878452 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.878423    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 23:23:50.879176 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.879151    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-m42kq\""
Sep 17 23:23:50.888222 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.887482    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hk6j7"]
Sep 17 23:23:50.894763 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.894319    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-spr8s"]
Sep 17 23:23:50.898800 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.898776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:50.908594 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.902519    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 23:23:50.908594 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.902782    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 23:23:50.908594 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.903006    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zslr5\""
Sep 17 23:23:50.908594 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:50.906204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-spr8s"]
Sep 17 23:23:51.014145 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.014121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/368951d8-63b6-4c36-a5ba-7c90ff58ee46-metrics-certs\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.014272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.014161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/368951d8-63b6-4c36-a5ba-7c90ff58ee46-cert\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.014272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.014219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f70af8b-63c6-4075-bbc7-bb909bdffb18-tls-certs\") pod \"model-serving-api-7b954ffc95-hk6j7\" (UID: \"6f70af8b-63c6-4075-bbc7-bb909bdffb18\") " pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:51.014272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.014235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npd9s\" (UniqueName: \"kubernetes.io/projected/6f70af8b-63c6-4075-bbc7-bb909bdffb18-kube-api-access-npd9s\") pod \"model-serving-api-7b954ffc95-hk6j7\" (UID: \"6f70af8b-63c6-4075-bbc7-bb909bdffb18\") " pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:51.014377 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.014299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dq9j\" (UniqueName: \"kubernetes.io/projected/368951d8-63b6-4c36-a5ba-7c90ff58ee46-kube-api-access-6dq9j\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.116522 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.116020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f70af8b-63c6-4075-bbc7-bb909bdffb18-tls-certs\") pod \"model-serving-api-7b954ffc95-hk6j7\" (UID: \"6f70af8b-63c6-4075-bbc7-bb909bdffb18\") " pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:51.116687 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.116545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd9s\" (UniqueName: \"kubernetes.io/projected/6f70af8b-63c6-4075-bbc7-bb909bdffb18-kube-api-access-npd9s\") pod \"model-serving-api-7b954ffc95-hk6j7\" (UID: \"6f70af8b-63c6-4075-bbc7-bb909bdffb18\") " pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:51.116687 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.116639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dq9j\" (UniqueName: \"kubernetes.io/projected/368951d8-63b6-4c36-a5ba-7c90ff58ee46-kube-api-access-6dq9j\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.117030 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.117008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/368951d8-63b6-4c36-a5ba-7c90ff58ee46-metrics-certs\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.117304 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.117286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/368951d8-63b6-4c36-a5ba-7c90ff58ee46-cert\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.123643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.123547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/368951d8-63b6-4c36-a5ba-7c90ff58ee46-cert\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.123782 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.123764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f70af8b-63c6-4075-bbc7-bb909bdffb18-tls-certs\") pod \"model-serving-api-7b954ffc95-hk6j7\" (UID: \"6f70af8b-63c6-4075-bbc7-bb909bdffb18\") " pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:51.124456 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.124437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/368951d8-63b6-4c36-a5ba-7c90ff58ee46-metrics-certs\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.126226 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.126208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dq9j\" (UniqueName: \"kubernetes.io/projected/368951d8-63b6-4c36-a5ba-7c90ff58ee46-kube-api-access-6dq9j\") pod \"odh-model-controller-7b966d7984-spr8s\" (UID: \"368951d8-63b6-4c36-a5ba-7c90ff58ee46\") " pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.126627 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.126605    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd9s\" (UniqueName: \"kubernetes.io/projected/6f70af8b-63c6-4075-bbc7-bb909bdffb18-kube-api-access-npd9s\") pod \"model-serving-api-7b954ffc95-hk6j7\" (UID: \"6f70af8b-63c6-4075-bbc7-bb909bdffb18\") " pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:51.194847 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.194830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:51.232133 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.232106    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:51.466626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.466554    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hk6j7"]
Sep 17 23:23:51.468972 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:23:51.468946    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f70af8b_63c6_4075_bbc7_bb909bdffb18.slice/crio-ce4a849a57ef7ab3c130b572b1edf3657d2a599d44c5c4487af2d0b919ca1027 WatchSource:0}: Error finding container ce4a849a57ef7ab3c130b572b1edf3657d2a599d44c5c4487af2d0b919ca1027: Status 404 returned error can't find the container with id ce4a849a57ef7ab3c130b572b1edf3657d2a599d44c5c4487af2d0b919ca1027
Sep 17 23:23:51.511124 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:51.511058    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-spr8s"]
Sep 17 23:23:51.512998 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:23:51.512975    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368951d8_63b6_4c36_a5ba_7c90ff58ee46.slice/crio-1b3b5e0bf5b61b4f59e6c451f282bf53841c08092de5ff84869957ef01d0557a WatchSource:0}: Error finding container 1b3b5e0bf5b61b4f59e6c451f282bf53841c08092de5ff84869957ef01d0557a: Status 404 returned error can't find the container with id 1b3b5e0bf5b61b4f59e6c451f282bf53841c08092de5ff84869957ef01d0557a
Sep 17 23:23:52.195323 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:52.195285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hk6j7" event={"ID":"6f70af8b-63c6-4075-bbc7-bb909bdffb18","Type":"ContainerStarted","Data":"ce4a849a57ef7ab3c130b572b1edf3657d2a599d44c5c4487af2d0b919ca1027"}
Sep 17 23:23:52.197539 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:52.197432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-spr8s" event={"ID":"368951d8-63b6-4c36-a5ba-7c90ff58ee46","Type":"ContainerStarted","Data":"1b3b5e0bf5b61b4f59e6c451f282bf53841c08092de5ff84869957ef01d0557a"}
Sep 17 23:23:55.215350 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:55.215320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hk6j7" event={"ID":"6f70af8b-63c6-4075-bbc7-bb909bdffb18","Type":"ContainerStarted","Data":"91da09e3b968b6024a4ea5bd333ad61f0dfb2bd80cfa24c4376442b36d74b4d5"}
Sep 17 23:23:55.215870 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:55.215420    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:23:55.217199 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:55.217179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-spr8s" event={"ID":"368951d8-63b6-4c36-a5ba-7c90ff58ee46","Type":"ContainerStarted","Data":"5d61296a8e3e28ac2feb25b8e37f0d44511b389a9b7915d507298e066c54e6d3"}
Sep 17 23:23:55.217308 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:55.217228    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:23:55.230966 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:55.230926    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-hk6j7" podStartSLOduration=1.567360326 podStartE2EDuration="5.230914657s" totalImagesPullingTime="3.663554331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:55.230375438 +0000 UTC m=+795.682206926" watchObservedRunningTime="2026-09-17 23:23:55.230914657 +0000 UTC m=+795.682746144"
Sep 17 23:23:55.247375 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:23:55.247336    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-spr8s" podStartSLOduration=1.664505019 podStartE2EDuration="5.247325397s" totalImagesPullingTime="3.582820378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:55.245542938 +0000 UTC m=+795.697374422" watchObservedRunningTime="2026-09-17 23:23:55.247325397 +0000 UTC m=+795.699156885"
Sep 17 23:24:06.225532 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:24:06.225448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-spr8s"
Sep 17 23:24:06.227800 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:24:06.227782    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-hk6j7"
Sep 17 23:25:40.018218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:40.018143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:25:40.021090 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:40.021067    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:25:40.026619 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:40.026598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:25:40.030887 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:40.030867    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:25:41.154310 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.154275    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"]
Sep 17 23:25:41.157105 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.157088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.160970 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.160947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 23:25:41.161101 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.161027    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 23:25:41.161163 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.161147    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-vsxfw\""
Sep 17 23:25:41.161163 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.161149    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-tz5d6\""
Sep 17 23:25:41.161260 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.161212    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 23:25:41.167320 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.167298    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"]
Sep 17 23:25:41.233453 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.233429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nsp\" (UniqueName: \"kubernetes.io/projected/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-kube-api-access-b2nsp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.233543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.233457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.334451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.334428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nsp\" (UniqueName: \"kubernetes.io/projected/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-kube-api-access-b2nsp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.334542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.334457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.336843 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.336827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.342975 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.342953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nsp\" (UniqueName: \"kubernetes.io/projected/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-kube-api-access-b2nsp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.468605 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.468527    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:41.741689 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.741659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"]
Sep 17 23:25:41.744703 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:25:41.744673    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc2c677_86aa_4fc0_81e8_9aff5a789ca9.slice/crio-f364590aaadbc475ac7271708d5ba90d36536f65e38ff9b7b612d87f488d33ce WatchSource:0}: Error finding container f364590aaadbc475ac7271708d5ba90d36536f65e38ff9b7b612d87f488d33ce: Status 404 returned error can't find the container with id f364590aaadbc475ac7271708d5ba90d36536f65e38ff9b7b612d87f488d33ce
Sep 17 23:25:41.747349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:41.747329    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:25:42.729886 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:42.729847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" event={"ID":"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9","Type":"ContainerStarted","Data":"f364590aaadbc475ac7271708d5ba90d36536f65e38ff9b7b612d87f488d33ce"}
Sep 17 23:25:44.743759 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:44.743724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" event={"ID":"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9","Type":"ContainerStarted","Data":"3754f31b4b9a8a6c27eb0abf002f734e0e187911dd35503c174c32dc234e4c31"}
Sep 17 23:25:44.744127 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:44.743806    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:25:44.759933 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:44.759885    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" podStartSLOduration=1.426659012 podStartE2EDuration="3.759871735s" totalImagesPullingTime="2.333212723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:44.758094215 +0000 UTC m=+905.209925707" watchObservedRunningTime="2026-09-17 23:25:44.759871735 +0000 UTC m=+905.211703220"
Sep 17 23:25:53.709259 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.709225    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"]
Sep 17 23:25:53.712816 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.712795    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.715293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.715271    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-w4pws\""
Sep 17 23:25:53.715404 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.715290    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 23:25:53.716549 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.716528    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 23:25:53.716641 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.716564    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 23:25:53.722554 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.722532    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"]
Sep 17 23:25:53.826809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.826788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.826901 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.826819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56330d86-9eea-470d-ac43-0bfb86fab49b-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.826901 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.826836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-home\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.826981 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.826924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.826981 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.826966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tqs7\" (UniqueName: \"kubernetes.io/projected/56330d86-9eea-470d-ac43-0bfb86fab49b-kube-api-access-7tqs7\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.827048 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.826993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-dshm\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.927958 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.927934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928088 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.927986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tqs7\" (UniqueName: \"kubernetes.io/projected/56330d86-9eea-470d-ac43-0bfb86fab49b-kube-api-access-7tqs7\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928088 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.928022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-dshm\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928088 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.928062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928276 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.928111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56330d86-9eea-470d-ac43-0bfb86fab49b-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928276 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.928151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-home\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928398 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.928377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.928379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.928522 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.928507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-home\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.930226 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.930204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-dshm\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.930468 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.930451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56330d86-9eea-470d-ac43-0bfb86fab49b-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:53.936112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:53.936094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tqs7\" (UniqueName: \"kubernetes.io/projected/56330d86-9eea-470d-ac43-0bfb86fab49b-kube-api-access-7tqs7\") pod \"fc-smoke-test-kserve-7f4687897-4r6tp\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:54.023775 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:54.023752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:25:54.294213 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:54.294057    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"]
Sep 17 23:25:54.297154 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:25:54.297126    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56330d86_9eea_470d_ac43_0bfb86fab49b.slice/crio-537dd652d532beed73313ccad0576462c744e889056fea6968d955a3efdd8dff WatchSource:0}: Error finding container 537dd652d532beed73313ccad0576462c744e889056fea6968d955a3efdd8dff: Status 404 returned error can't find the container with id 537dd652d532beed73313ccad0576462c744e889056fea6968d955a3efdd8dff
Sep 17 23:25:54.789719 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:54.789681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" event={"ID":"56330d86-9eea-470d-ac43-0bfb86fab49b","Type":"ContainerStarted","Data":"537dd652d532beed73313ccad0576462c744e889056fea6968d955a3efdd8dff"}
Sep 17 23:25:56.800278 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:56.800242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" event={"ID":"56330d86-9eea-470d-ac43-0bfb86fab49b","Type":"ContainerStarted","Data":"b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e"}
Sep 17 23:25:56.818264 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:56.818221    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" podStartSLOduration=2.312317576 podStartE2EDuration="3.818206211s" totalImagesPullingTime="1.505888635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:56.815987094 +0000 UTC m=+917.267818582" watchObservedRunningTime="2026-09-17 23:25:56.818206211 +0000 UTC m=+917.270037730"
Sep 17 23:25:57.023946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:25:57.023914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:26:04.024072 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:04.024042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:26:04.036173 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:04.036147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:26:04.855254 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:04.855222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:26:15.750095 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:15.750060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:26:17.347269 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:17.346647    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"]
Sep 17 23:26:25.749711 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:25.749663    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" podUID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:26:28.346564 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.346525    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 23:26:28.346564 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:26:28.346565    2566 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" podUID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" containerName="main" containerID="cri-o://3754f31b4b9a8a6c27eb0abf002f734e0e187911dd35503c174c32dc234e4c31"
Sep 17 23:26:28.346956 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.346639    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" podUID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" containerName="main" containerID="cri-o://3754f31b4b9a8a6c27eb0abf002f734e0e187911dd35503c174c32dc234e4c31" gracePeriod=50
Sep 17 23:26:28.970144 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.970109    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd_2dc2c677-86aa-4fc0-81e8-9aff5a789ca9/main/0.log"
Sep 17 23:26:28.970279 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.970167    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" containerID="3754f31b4b9a8a6c27eb0abf002f734e0e187911dd35503c174c32dc234e4c31" exitCode=1
Sep 17 23:26:28.970279 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.970240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" event={"ID":"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9","Type":"ContainerDied","Data":"3754f31b4b9a8a6c27eb0abf002f734e0e187911dd35503c174c32dc234e4c31"}
Sep 17 23:26:28.970369 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.970305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd" event={"ID":"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9","Type":"ContainerDied","Data":"f364590aaadbc475ac7271708d5ba90d36536f65e38ff9b7b612d87f488d33ce"}
Sep 17 23:26:28.970369 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.970333    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f364590aaadbc475ac7271708d5ba90d36536f65e38ff9b7b612d87f488d33ce"
Sep 17 23:26:28.982055 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.982028    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd_2dc2c677-86aa-4fc0-81e8-9aff5a789ca9/main/0.log"
Sep 17 23:26:28.984365 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:28.984351    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:26:29.099766 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.099741    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2nsp\" (UniqueName: \"kubernetes.io/projected/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-kube-api-access-b2nsp\") pod \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") "
Sep 17 23:26:29.099886 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.099804    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-tls-certs\") pod \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\" (UID: \"2dc2c677-86aa-4fc0-81e8-9aff5a789ca9\") "
Sep 17 23:26:29.101833 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.101812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" (UID: "2dc2c677-86aa-4fc0-81e8-9aff5a789ca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:26:29.101948 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.101934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-kube-api-access-b2nsp" (OuterVolumeSpecName: "kube-api-access-b2nsp") pod "2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" (UID: "2dc2c677-86aa-4fc0-81e8-9aff5a789ca9"). InnerVolumeSpecName "kube-api-access-b2nsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:26:29.201250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.201225    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2nsp\" (UniqueName: \"kubernetes.io/projected/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-kube-api-access-b2nsp\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:29.201250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.201247    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:29.974733 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.974700    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"
Sep 17 23:26:29.995047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:29.995022    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"]
Sep 17 23:26:30.001283 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:30.001259    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kdvhd"]
Sep 17 23:26:30.013535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:30.013511    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" path="/var/lib/kubelet/pods/2dc2c677-86aa-4fc0-81e8-9aff5a789ca9/volumes"
Sep 17 23:26:33.166302 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.166252    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"]
Sep 17 23:26:33.178736 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:26:33.178698    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:26:33.178736 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:26:33.178736 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:26:33.178736 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e" cmd=["/bin/sleep","15"]
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.178752    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" message="" err=<
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:26:33.178765    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" podUID="56330d86-9eea-470d-ac43-0bfb86fab49b" containerName="main" containerID="cri-o://b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e"
Sep 17 23:26:33.179061 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.178807    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" podUID="56330d86-9eea-470d-ac43-0bfb86fab49b" containerName="main" containerID="cri-o://b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e" gracePeriod=60
Sep 17 23:26:33.542112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.542078    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:26:33.634465 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634437    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-home\") pod \"56330d86-9eea-470d-ac43-0bfb86fab49b\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") "
Sep 17 23:26:33.634645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56330d86-9eea-470d-ac43-0bfb86fab49b-tls-certs\") pod \"56330d86-9eea-470d-ac43-0bfb86fab49b\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") "
Sep 17 23:26:33.634645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634503    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tqs7\" (UniqueName: \"kubernetes.io/projected/56330d86-9eea-470d-ac43-0bfb86fab49b-kube-api-access-7tqs7\") pod \"56330d86-9eea-470d-ac43-0bfb86fab49b\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") "
Sep 17 23:26:33.634645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-dshm\") pod \"56330d86-9eea-470d-ac43-0bfb86fab49b\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") "
Sep 17 23:26:33.634645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-model-cache\") pod \"56330d86-9eea-470d-ac43-0bfb86fab49b\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") "
Sep 17 23:26:33.634785 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-tmp-dir\") pod \"56330d86-9eea-470d-ac43-0bfb86fab49b\" (UID: \"56330d86-9eea-470d-ac43-0bfb86fab49b\") "
Sep 17 23:26:33.634785 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-home" (OuterVolumeSpecName: "home") pod "56330d86-9eea-470d-ac43-0bfb86fab49b" (UID: "56330d86-9eea-470d-ac43-0bfb86fab49b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:33.634785 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-model-cache" (OuterVolumeSpecName: "model-cache") pod "56330d86-9eea-470d-ac43-0bfb86fab49b" (UID: "56330d86-9eea-470d-ac43-0bfb86fab49b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:33.634991 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634972    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:33.635046 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634997    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:33.635046 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.634994    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56330d86-9eea-470d-ac43-0bfb86fab49b" (UID: "56330d86-9eea-470d-ac43-0bfb86fab49b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:33.636739 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.636713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56330d86-9eea-470d-ac43-0bfb86fab49b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56330d86-9eea-470d-ac43-0bfb86fab49b" (UID: "56330d86-9eea-470d-ac43-0bfb86fab49b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:26:33.637001 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.636980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56330d86-9eea-470d-ac43-0bfb86fab49b-kube-api-access-7tqs7" (OuterVolumeSpecName: "kube-api-access-7tqs7") pod "56330d86-9eea-470d-ac43-0bfb86fab49b" (UID: "56330d86-9eea-470d-ac43-0bfb86fab49b"). InnerVolumeSpecName "kube-api-access-7tqs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:26:33.637001 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.636981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-dshm" (OuterVolumeSpecName: "dshm") pod "56330d86-9eea-470d-ac43-0bfb86fab49b" (UID: "56330d86-9eea-470d-ac43-0bfb86fab49b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:33.735610 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.735568    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:33.735711 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.735612    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56330d86-9eea-470d-ac43-0bfb86fab49b-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:33.735711 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.735627    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tqs7\" (UniqueName: \"kubernetes.io/projected/56330d86-9eea-470d-ac43-0bfb86fab49b-kube-api-access-7tqs7\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:33.735711 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.735636    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56330d86-9eea-470d-ac43-0bfb86fab49b-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:26:33.994526 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.994454    2566 generic.go:356] "Generic (PLEG): container finished" podID="56330d86-9eea-470d-ac43-0bfb86fab49b" containerID="b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e" exitCode=0
Sep 17 23:26:33.994526 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.994510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" event={"ID":"56330d86-9eea-470d-ac43-0bfb86fab49b","Type":"ContainerDied","Data":"b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e"}
Sep 17 23:26:33.994752 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.994562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp" event={"ID":"56330d86-9eea-470d-ac43-0bfb86fab49b","Type":"ContainerDied","Data":"537dd652d532beed73313ccad0576462c744e889056fea6968d955a3efdd8dff"}
Sep 17 23:26:33.994752 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.994591    2566 scope.go:117] "RemoveContainer" containerID="b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e"
Sep 17 23:26:33.994752 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:33.994515    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"
Sep 17 23:26:34.004436 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:34.004416    2566 scope.go:117] "RemoveContainer" containerID="b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e"
Sep 17 23:26:34.004784 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:26:34.004758    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e\": container with ID starting with b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e not found: ID does not exist" containerID="b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e"
Sep 17 23:26:34.004864 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:34.004797    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e"} err="failed to get container status \"b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e\": rpc error: code = NotFound desc = could not find container \"b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e\": container with ID starting with b2d2e418214d085d421a505acd6cf4033650e7a6787133a569ec7def9856ed1e not found: ID does not exist"
Sep 17 23:26:34.016012 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:34.015983    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"]
Sep 17 23:26:34.017906 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:34.017886    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-4r6tp"]
Sep 17 23:26:36.014518 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:26:36.014477    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56330d86-9eea-470d-ac43-0bfb86fab49b" path="/var/lib/kubelet/pods/56330d86-9eea-470d-ac43-0bfb86fab49b/volumes"
Sep 17 23:27:10.588717 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.588628    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"]
Sep 17 23:27:10.589288 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.589028    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" containerName="main"
Sep 17 23:27:10.589288 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.589043    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" containerName="main"
Sep 17 23:27:10.589288 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.589065    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56330d86-9eea-470d-ac43-0bfb86fab49b" containerName="main"
Sep 17 23:27:10.589288 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.589071    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56330d86-9eea-470d-ac43-0bfb86fab49b" containerName="main"
Sep 17 23:27:10.589288 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.589139    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dc2c677-86aa-4fc0-81e8-9aff5a789ca9" containerName="main"
Sep 17 23:27:10.589288 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.589149    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="56330d86-9eea-470d-ac43-0bfb86fab49b" containerName="main"
Sep 17 23:27:10.592293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.592271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.596490 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.596465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 23:27:10.596668 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.596478    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-bxngh\""
Sep 17 23:27:10.596914 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.596893    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 23:27:10.597025 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.596595    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 23:27:10.597025 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.596511    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-w74nv\""
Sep 17 23:27:10.602515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.602485    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"]
Sep 17 23:27:10.704285 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.704261    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.704393 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.704302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.704393 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.704320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f4q\" (UniqueName: \"kubernetes.io/projected/3fa2c527-5507-401a-886b-c3e5dd3a95a6-kube-api-access-w5f4q\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.704474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.704443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.805771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.805749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.805890 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.805785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.805890 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.805810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f4q\" (UniqueName: \"kubernetes.io/projected/3fa2c527-5507-401a-886b-c3e5dd3a95a6-kube-api-access-w5f4q\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.805890 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.805867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.806150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.806125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.806218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.806200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.808177 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.808160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.813054 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.813034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f4q\" (UniqueName: \"kubernetes.io/projected/3fa2c527-5507-401a-886b-c3e5dd3a95a6-kube-api-access-w5f4q\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:10.904765 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:10.904713    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:11.179870 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:11.179796    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"]
Sep 17 23:27:11.182929 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:27:11.182902    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa2c527_5507_401a_886b_c3e5dd3a95a6.slice/crio-094dfc183f8c0669b823b1d056fe1eab5c7dc3bfa82bb1b029d4d155f3ca6ede WatchSource:0}: Error finding container 094dfc183f8c0669b823b1d056fe1eab5c7dc3bfa82bb1b029d4d155f3ca6ede: Status 404 returned error can't find the container with id 094dfc183f8c0669b823b1d056fe1eab5c7dc3bfa82bb1b029d4d155f3ca6ede
Sep 17 23:27:12.182051 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:12.182020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" event={"ID":"3fa2c527-5507-401a-886b-c3e5dd3a95a6","Type":"ContainerStarted","Data":"c936185bedb97ebe9e92b0f11b61c1daac3542f9bf55f53331fbe08291862760"}
Sep 17 23:27:12.182407 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:12.182070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" event={"ID":"3fa2c527-5507-401a-886b-c3e5dd3a95a6","Type":"ContainerStarted","Data":"094dfc183f8c0669b823b1d056fe1eab5c7dc3bfa82bb1b029d4d155f3ca6ede"}
Sep 17 23:27:12.182407 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:12.182117    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:12.216492 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:12.216440    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" podStartSLOduration=2.216426352 podStartE2EDuration="2.216426352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:12.215962964 +0000 UTC m=+992.667794451" watchObservedRunningTime="2026-09-17 23:27:12.216426352 +0000 UTC m=+992.668257863"
Sep 17 23:27:29.054330 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.054292    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"]
Sep 17 23:27:29.058332 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.058307    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.066873 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.066841    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 23:27:29.068205 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.068143    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 23:27:29.068328 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.068265    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 23:27:29.068390 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.068175    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-mdg29\""
Sep 17 23:27:29.068442 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.068175    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-qnjt9\""
Sep 17 23:27:29.070880 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.070856    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"]
Sep 17 23:27:29.148347 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.148320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a328b44-41a6-439c-83d0-7412d4188c14-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.148442 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.148378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4cr\" (UniqueName: \"kubernetes.io/projected/0a328b44-41a6-439c-83d0-7412d4188c14-kube-api-access-tp4cr\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.249244 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.249219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a328b44-41a6-439c-83d0-7412d4188c14-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.249361 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.249284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4cr\" (UniqueName: \"kubernetes.io/projected/0a328b44-41a6-439c-83d0-7412d4188c14-kube-api-access-tp4cr\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.251626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.251600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a328b44-41a6-439c-83d0-7412d4188c14-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.257619 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.257597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4cr\" (UniqueName: \"kubernetes.io/projected/0a328b44-41a6-439c-83d0-7412d4188c14-kube-api-access-tp4cr\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.377724 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.377664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:29.648258 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:29.648227    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"]
Sep 17 23:27:29.650419 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:27:29.650388    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a328b44_41a6_439c_83d0_7412d4188c14.slice/crio-e988057a2d76a047c636217cef681d9e0d733c5a0dab3c22db89fce532a08896 WatchSource:0}: Error finding container e988057a2d76a047c636217cef681d9e0d733c5a0dab3c22db89fce532a08896: Status 404 returned error can't find the container with id e988057a2d76a047c636217cef681d9e0d733c5a0dab3c22db89fce532a08896
Sep 17 23:27:30.270938 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:30.270905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" event={"ID":"0a328b44-41a6-439c-83d0-7412d4188c14","Type":"ContainerStarted","Data":"18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd"}
Sep 17 23:27:30.271342 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:30.270958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" event={"ID":"0a328b44-41a6-439c-83d0-7412d4188c14","Type":"ContainerStarted","Data":"e988057a2d76a047c636217cef681d9e0d733c5a0dab3c22db89fce532a08896"}
Sep 17 23:27:30.271342 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:30.271005    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:27:30.291087 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:30.291046    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" podStartSLOduration=1.291034529 podStartE2EDuration="1.291034529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:30.289345922 +0000 UTC m=+1010.741177411" watchObservedRunningTime="2026-09-17 23:27:30.291034529 +0000 UTC m=+1010.742866017"
Sep 17 23:27:43.188541 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:43.188508    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:27:49.420471 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:49.420434    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"]
Sep 17 23:27:53.189122 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:27:53.189083    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:01.277516 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:01.277481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:28:03.189404 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:03.189360    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:04.447457 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:04.447408    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main" containerID="cri-o://c936185bedb97ebe9e92b0f11b61c1daac3542f9bf55f53331fbe08291862760" gracePeriod=45
Sep 17 23:28:08.325838 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:08.325802    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"]
Sep 17 23:28:11.278230 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:11.278181    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:13.189250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:13.189209    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:13.189681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:13.189310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:28:21.277533 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:21.277482    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:23.188961 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:23.188926    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:23.352465 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:23.352432    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main" containerID="cri-o://18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd" gracePeriod=45
Sep 17 23:28:31.277596 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:31.277538    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:31.278015 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:31.277689    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:28:33.188765 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:33.188670    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:34.585875 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.585838    2566 generic.go:356] "Generic (PLEG): container finished" podID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerID="c936185bedb97ebe9e92b0f11b61c1daac3542f9bf55f53331fbe08291862760" exitCode=0
Sep 17 23:28:34.586210 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.585903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" event={"ID":"3fa2c527-5507-401a-886b-c3e5dd3a95a6","Type":"ContainerDied","Data":"c936185bedb97ebe9e92b0f11b61c1daac3542f9bf55f53331fbe08291862760"}
Sep 17 23:28:34.705150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.705103    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:28:34.750516 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.750495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5f4q\" (UniqueName: \"kubernetes.io/projected/3fa2c527-5507-401a-886b-c3e5dd3a95a6-kube-api-access-w5f4q\") pod \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") "
Sep 17 23:28:34.750685 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.750551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tokenizer-cache\") pod \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") "
Sep 17 23:28:34.750685 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.750618    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-cachi2-cache\") pod \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") "
Sep 17 23:28:34.750685 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.750644    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tls-certs\") pod \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\" (UID: \"3fa2c527-5507-401a-886b-c3e5dd3a95a6\") "
Sep 17 23:28:34.750858 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.750840    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "3fa2c527-5507-401a-886b-c3e5dd3a95a6" (UID: "3fa2c527-5507-401a-886b-c3e5dd3a95a6"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:34.750914 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.750862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3fa2c527-5507-401a-886b-c3e5dd3a95a6" (UID: "3fa2c527-5507-401a-886b-c3e5dd3a95a6"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:34.752615 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.752591    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa2c527-5507-401a-886b-c3e5dd3a95a6-kube-api-access-w5f4q" (OuterVolumeSpecName: "kube-api-access-w5f4q") pod "3fa2c527-5507-401a-886b-c3e5dd3a95a6" (UID: "3fa2c527-5507-401a-886b-c3e5dd3a95a6"). InnerVolumeSpecName "kube-api-access-w5f4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:34.752693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.752595    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fa2c527-5507-401a-886b-c3e5dd3a95a6" (UID: "3fa2c527-5507-401a-886b-c3e5dd3a95a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:28:34.852192 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.852165    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f4q\" (UniqueName: \"kubernetes.io/projected/3fa2c527-5507-401a-886b-c3e5dd3a95a6-kube-api-access-w5f4q\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:28:34.852192 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.852189    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tokenizer-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:28:34.852306 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.852202    2566 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa2c527-5507-401a-886b-c3e5dd3a95a6-cachi2-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:28:34.852306 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:34.852215    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa2c527-5507-401a-886b-c3e5dd3a95a6-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:28:35.592987 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:35.592955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh" event={"ID":"3fa2c527-5507-401a-886b-c3e5dd3a95a6","Type":"ContainerDied","Data":"094dfc183f8c0669b823b1d056fe1eab5c7dc3bfa82bb1b029d4d155f3ca6ede"}
Sep 17 23:28:35.593474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:35.592999    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"
Sep 17 23:28:35.593474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:35.593018    2566 scope.go:117] "RemoveContainer" containerID="c936185bedb97ebe9e92b0f11b61c1daac3542f9bf55f53331fbe08291862760"
Sep 17 23:28:35.615209 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:35.615179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"]
Sep 17 23:28:35.616877 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:35.616853    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc8d6mh"]
Sep 17 23:28:36.014068 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:36.014031    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" path="/var/lib/kubelet/pods/3fa2c527-5507-401a-886b-c3e5dd3a95a6/volumes"
Sep 17 23:28:41.277536 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:41.277491    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:43.325509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.325467    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"]
Sep 17 23:28:43.326047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.326026    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main"
Sep 17 23:28:43.326090 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.326051    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main"
Sep 17 23:28:43.326171 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.326159    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa2c527-5507-401a-886b-c3e5dd3a95a6" containerName="main"
Sep 17 23:28:43.330385 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.330368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.334935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.334901    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-d5swj\""
Sep 17 23:28:43.335066 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.334939    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 23:28:43.335066 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.335042    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 23:28:43.335217 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.335194    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-jgn68\""
Sep 17 23:28:43.335346 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.335244    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 23:28:43.335743 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.335721    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"]
Sep 17 23:28:43.414680 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.414651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx22d\" (UniqueName: \"kubernetes.io/projected/620818af-b6f2-438d-bb10-c6783655c662-kube-api-access-cx22d\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.414779 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.414696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620818af-b6f2-438d-bb10-c6783655c662-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.515304 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.515280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620818af-b6f2-438d-bb10-c6783655c662-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.515401 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.515340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx22d\" (UniqueName: \"kubernetes.io/projected/620818af-b6f2-438d-bb10-c6783655c662-kube-api-access-cx22d\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.517763 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.517736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620818af-b6f2-438d-bb10-c6783655c662-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.523946 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.523923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx22d\" (UniqueName: \"kubernetes.io/projected/620818af-b6f2-438d-bb10-c6783655c662-kube-api-access-cx22d\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.642000 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.641926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:43.920068 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:43.920044    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"]
Sep 17 23:28:43.922946 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:28:43.922910    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620818af_b6f2_438d_bb10_c6783655c662.slice/crio-47bc7da10fc3df4f16116524e31737baa1b67c5a385402ebf3e1dca6ff11ff44 WatchSource:0}: Error finding container 47bc7da10fc3df4f16116524e31737baa1b67c5a385402ebf3e1dca6ff11ff44: Status 404 returned error can't find the container with id 47bc7da10fc3df4f16116524e31737baa1b67c5a385402ebf3e1dca6ff11ff44
Sep 17 23:28:44.641341 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:44.641308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" event={"ID":"620818af-b6f2-438d-bb10-c6783655c662","Type":"ContainerStarted","Data":"00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a"}
Sep 17 23:28:44.641728 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:44.641363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" event={"ID":"620818af-b6f2-438d-bb10-c6783655c662","Type":"ContainerStarted","Data":"47bc7da10fc3df4f16116524e31737baa1b67c5a385402ebf3e1dca6ff11ff44"}
Sep 17 23:28:44.641728 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:44.641388    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:28:44.659087 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:44.659042    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" podStartSLOduration=1.6590278550000002 podStartE2EDuration="1.659027855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:44.658160995 +0000 UTC m=+1085.109992484" watchObservedRunningTime="2026-09-17 23:28:44.659027855 +0000 UTC m=+1085.110859343"
Sep 17 23:28:51.277729 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:51.277684    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:28:53.591659 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.591636    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:28:53.687423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.687359    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a328b44-41a6-439c-83d0-7412d4188c14" containerID="18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd" exitCode=0
Sep 17 23:28:53.687423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.687396    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"
Sep 17 23:28:53.687624 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.687443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" event={"ID":"0a328b44-41a6-439c-83d0-7412d4188c14","Type":"ContainerDied","Data":"18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd"}
Sep 17 23:28:53.687624 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.687487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd" event={"ID":"0a328b44-41a6-439c-83d0-7412d4188c14","Type":"ContainerDied","Data":"e988057a2d76a047c636217cef681d9e0d733c5a0dab3c22db89fce532a08896"}
Sep 17 23:28:53.687624 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.687511    2566 scope.go:117] "RemoveContainer" containerID="18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd"
Sep 17 23:28:53.695464 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.695446    2566 scope.go:117] "RemoveContainer" containerID="18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd"
Sep 17 23:28:53.695522 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.695473    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a328b44-41a6-439c-83d0-7412d4188c14-tls-certs\") pod \"0a328b44-41a6-439c-83d0-7412d4188c14\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") "
Sep 17 23:28:53.695602 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.695585    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp4cr\" (UniqueName: \"kubernetes.io/projected/0a328b44-41a6-439c-83d0-7412d4188c14-kube-api-access-tp4cr\") pod \"0a328b44-41a6-439c-83d0-7412d4188c14\" (UID: \"0a328b44-41a6-439c-83d0-7412d4188c14\") "
Sep 17 23:28:53.695842 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:28:53.695816    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd\": container with ID starting with 18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd not found: ID does not exist" containerID="18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd"
Sep 17 23:28:53.695942 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.695844    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd"} err="failed to get container status \"18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd\": rpc error: code = NotFound desc = could not find container \"18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd\": container with ID starting with 18f6c6c6637e2f7df47df6bc8eb3e1207ea72633d53dce6f91de5a852231adcd not found: ID does not exist"
Sep 17 23:28:53.697441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.697417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a328b44-41a6-439c-83d0-7412d4188c14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a328b44-41a6-439c-83d0-7412d4188c14" (UID: "0a328b44-41a6-439c-83d0-7412d4188c14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:28:53.697531 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.697516    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a328b44-41a6-439c-83d0-7412d4188c14-kube-api-access-tp4cr" (OuterVolumeSpecName: "kube-api-access-tp4cr") pod "0a328b44-41a6-439c-83d0-7412d4188c14" (UID: "0a328b44-41a6-439c-83d0-7412d4188c14"). InnerVolumeSpecName "kube-api-access-tp4cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:53.796800 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.796777    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp4cr\" (UniqueName: \"kubernetes.io/projected/0a328b44-41a6-439c-83d0-7412d4188c14-kube-api-access-tp4cr\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:28:53.796800 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:53.796796    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a328b44-41a6-439c-83d0-7412d4188c14-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:28:54.014523 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:54.014499    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"]
Sep 17 23:28:54.014642 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:54.014532    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqhnd"]
Sep 17 23:28:56.014458 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:28:56.014421    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" path="/var/lib/kubelet/pods/0a328b44-41a6-439c-83d0-7412d4188c14/volumes"
Sep 17 23:29:11.185474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.185434    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"]
Sep 17 23:29:11.185867 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.185796    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main"
Sep 17 23:29:11.185867 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.185808    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main"
Sep 17 23:29:11.185936 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.185887    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a328b44-41a6-439c-83d0-7412d4188c14" containerName="main"
Sep 17 23:29:11.188862 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.188841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.193263 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.193239    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 23:29:11.193263 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.193260    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 23:29:11.193452 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.193332    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 23:29:11.193452 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.193347    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-bg9hb\""
Sep 17 23:29:11.199212 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.199184    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"]
Sep 17 23:29:11.332407 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.332383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.332515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.332410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.332515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.332441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.332515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.332474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.332515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.332498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrqn\" (UniqueName: \"kubernetes.io/projected/dc428031-0351-4b2d-a3b4-9c7f81c21321-kube-api-access-fwrqn\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.332684 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.332583    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433444 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433750 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrqn\" (UniqueName: \"kubernetes.io/projected/dc428031-0351-4b2d-a3b4-9c7f81c21321-kube-api-access-fwrqn\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433750 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433867 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433933 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.433984 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.433950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.435847 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.435799    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.436009 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.435992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.441328 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.441301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrqn\" (UniqueName: \"kubernetes.io/projected/dc428031-0351-4b2d-a3b4-9c7f81c21321-kube-api-access-fwrqn\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.501984 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.501963    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:11.782880 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.782685    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"]
Sep 17 23:29:11.785241 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:29:11.785215    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc428031_0351_4b2d_a3b4_9c7f81c21321.slice/crio-0d830463dbbc5e3bb0f7a7723e50e52aeb4cf9016173043a3bceef3b56f11451 WatchSource:0}: Error finding container 0d830463dbbc5e3bb0f7a7723e50e52aeb4cf9016173043a3bceef3b56f11451: Status 404 returned error can't find the container with id 0d830463dbbc5e3bb0f7a7723e50e52aeb4cf9016173043a3bceef3b56f11451
Sep 17 23:29:11.793094 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:11.792915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" event={"ID":"dc428031-0351-4b2d-a3b4-9c7f81c21321","Type":"ContainerStarted","Data":"0d830463dbbc5e3bb0f7a7723e50e52aeb4cf9016173043a3bceef3b56f11451"}
Sep 17 23:29:12.799130 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:12.799098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" event={"ID":"dc428031-0351-4b2d-a3b4-9c7f81c21321","Type":"ContainerStarted","Data":"f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd"}
Sep 17 23:29:12.816590 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:12.816527    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" podStartSLOduration=1.816512345 podStartE2EDuration="1.816512345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:12.814535221 +0000 UTC m=+1113.266366713" watchObservedRunningTime="2026-09-17 23:29:12.816512345 +0000 UTC m=+1113.268343832"
Sep 17 23:29:13.502749 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:13.502699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:14.703656 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:14.703621    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"]
Sep 17 23:29:14.964468 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:14.964380    2566 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 23:29:14.964468 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:14.964444    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs podName:dc428031-0351-4b2d-a3b4-9c7f81c21321 nodeName:}" failed. No retries permitted until 2026-09-17 23:29:15.464428773 +0000 UTC m=+1115.916260253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 23:29:15.469378 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:15.469347    2566 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 23:29:15.469544 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:15.469415    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs podName:dc428031-0351-4b2d-a3b4-9c7f81c21321 nodeName:}" failed. No retries permitted until 2026-09-17 23:29:16.469400915 +0000 UTC m=+1116.921232380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 23:29:15.649333 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:15.649305    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:29:15.822166 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:15.822136    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:29:15.822166 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:15.822166 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:15.822166 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd" cmd=["/bin/sleep","15"]
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:15.822178    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" message="" err=<
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:15.822192    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" podUID="dc428031-0351-4b2d-a3b4-9c7f81c21321" containerName="main" containerID="cri-o://f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd"
Sep 17 23:29:15.822610 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:15.822234    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" podUID="dc428031-0351-4b2d-a3b4-9c7f81c21321" containerName="main" containerID="cri-o://f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd" gracePeriod=60
Sep 17 23:29:16.175398 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.175351    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:16.276805 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.276773    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs\") pod \"dc428031-0351-4b2d-a3b4-9c7f81c21321\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") "
Sep 17 23:29:16.276979 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.276826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-home\") pod \"dc428031-0351-4b2d-a3b4-9c7f81c21321\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") "
Sep 17 23:29:16.277052 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.276990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwrqn\" (UniqueName: \"kubernetes.io/projected/dc428031-0351-4b2d-a3b4-9c7f81c21321-kube-api-access-fwrqn\") pod \"dc428031-0351-4b2d-a3b4-9c7f81c21321\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") "
Sep 17 23:29:16.277052 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277024    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-model-cache\") pod \"dc428031-0351-4b2d-a3b4-9c7f81c21321\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") "
Sep 17 23:29:16.277167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-tmp-dir\") pod \"dc428031-0351-4b2d-a3b4-9c7f81c21321\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") "
Sep 17 23:29:16.277167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-home" (OuterVolumeSpecName: "home") pod "dc428031-0351-4b2d-a3b4-9c7f81c21321" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:16.277167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277095    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-dshm\") pod \"dc428031-0351-4b2d-a3b4-9c7f81c21321\" (UID: \"dc428031-0351-4b2d-a3b4-9c7f81c21321\") "
Sep 17 23:29:16.277311 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277280    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc428031-0351-4b2d-a3b4-9c7f81c21321" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:16.277311 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277293    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc428031-0351-4b2d-a3b4-9c7f81c21321" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:16.277447 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277418    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.277447 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277437    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.277447 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.277446    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.279041 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.279018    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-dshm" (OuterVolumeSpecName: "dshm") pod "dc428031-0351-4b2d-a3b4-9c7f81c21321" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:16.279160 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.279087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc428031-0351-4b2d-a3b4-9c7f81c21321-kube-api-access-fwrqn" (OuterVolumeSpecName: "kube-api-access-fwrqn") pod "dc428031-0351-4b2d-a3b4-9c7f81c21321" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321"). InnerVolumeSpecName "kube-api-access-fwrqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:29:16.279160 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.279087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc428031-0351-4b2d-a3b4-9c7f81c21321" (UID: "dc428031-0351-4b2d-a3b4-9c7f81c21321"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:29:16.378894 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.378826    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc428031-0351-4b2d-a3b4-9c7f81c21321-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.378894 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.378853    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrqn\" (UniqueName: \"kubernetes.io/projected/dc428031-0351-4b2d-a3b4-9c7f81c21321-kube-api-access-fwrqn\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.378894 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.378862    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc428031-0351-4b2d-a3b4-9c7f81c21321-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.819827 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.819796    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc428031-0351-4b2d-a3b4-9c7f81c21321" containerID="f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd" exitCode=0
Sep 17 23:29:16.819988 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.819841    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"
Sep 17 23:29:16.819988 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.819879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" event={"ID":"dc428031-0351-4b2d-a3b4-9c7f81c21321","Type":"ContainerDied","Data":"f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd"}
Sep 17 23:29:16.819988 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.819935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t" event={"ID":"dc428031-0351-4b2d-a3b4-9c7f81c21321","Type":"ContainerDied","Data":"0d830463dbbc5e3bb0f7a7723e50e52aeb4cf9016173043a3bceef3b56f11451"}
Sep 17 23:29:16.819988 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.819963    2566 scope.go:117] "RemoveContainer" containerID="f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd"
Sep 17 23:29:16.829061 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.828912    2566 scope.go:117] "RemoveContainer" containerID="f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd"
Sep 17 23:29:16.829270 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:16.829165    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd\": container with ID starting with f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd not found: ID does not exist" containerID="f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd"
Sep 17 23:29:16.829270 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.829189    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd"} err="failed to get container status \"f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd\": rpc error: code = NotFound desc = could not find container \"f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd\": container with ID starting with f237ce416cb2994ba0333f7f990c32cba17db3e0ef736e8e6c3eafb844e905cd not found: ID does not exist"
Sep 17 23:29:16.841167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.841147    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"]
Sep 17 23:29:16.844590 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:16.844550    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-q675t"]
Sep 17 23:29:18.014157 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:18.014128    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc428031-0351-4b2d-a3b4-9c7f81c21321" path="/var/lib/kubelet/pods/dc428031-0351-4b2d-a3b4-9c7f81c21321/volumes"
Sep 17 23:29:29.194672 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.194631    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"]
Sep 17 23:29:29.195365 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.194971    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc428031-0351-4b2d-a3b4-9c7f81c21321" containerName="main"
Sep 17 23:29:29.195365 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.194982    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc428031-0351-4b2d-a3b4-9c7f81c21321" containerName="main"
Sep 17 23:29:29.195365 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.195083    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc428031-0351-4b2d-a3b4-9c7f81c21321" containerName="main"
Sep 17 23:29:29.199698 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.199670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.203974 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.203946    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 23:29:29.204194 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.203985    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 23:29:29.204340 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.203989    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-p7ctd\""
Sep 17 23:29:29.204465 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.204019    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 23:29:29.209824 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.209803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"]
Sep 17 23:29:29.279009 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.278986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.279135 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.279020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8rg\" (UniqueName: \"kubernetes.io/projected/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-kube-api-access-6v8rg\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.279135 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.279109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.279214 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.279138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.279214 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.279164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.279447 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.279220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380159 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380258 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380258 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380258 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380366 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8rg\" (UniqueName: \"kubernetes.io/projected/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-kube-api-access-6v8rg\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380638 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380723 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.380723 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.380695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.382301 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.382284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.382729 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.382710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.388151 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.388131    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8rg\" (UniqueName: \"kubernetes.io/projected/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-kube-api-access-6v8rg\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.512107 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.512087    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:29.785204 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.785118    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"]
Sep 17 23:29:29.789194 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:29:29.789163    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bdb8a0_cacf_4fca_8ea7_e3ee8ce7e2aa.slice/crio-0645ecb96ce82a6ec5a02caa57561c3d49af20ac3738e5155d83134aaf3adac2 WatchSource:0}: Error finding container 0645ecb96ce82a6ec5a02caa57561c3d49af20ac3738e5155d83134aaf3adac2: Status 404 returned error can't find the container with id 0645ecb96ce82a6ec5a02caa57561c3d49af20ac3738e5155d83134aaf3adac2
Sep 17 23:29:29.888586 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.888524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" event={"ID":"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa","Type":"ContainerStarted","Data":"f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9"}
Sep 17 23:29:29.888738 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.888625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" event={"ID":"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa","Type":"ContainerStarted","Data":"0645ecb96ce82a6ec5a02caa57561c3d49af20ac3738e5155d83134aaf3adac2"}
Sep 17 23:29:29.906256 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:29.906204    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" podStartSLOduration=0.906189578 podStartE2EDuration="906.189578ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:29.904118504 +0000 UTC m=+1130.355949992" watchObservedRunningTime="2026-09-17 23:29:29.906189578 +0000 UTC m=+1130.358021066"
Sep 17 23:29:30.512467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:30.512437    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:32.031280 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:32.031233    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"]
Sep 17 23:29:33.117481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.117444    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"]
Sep 17 23:29:33.126523 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:33.126492    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:29:33.126523 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:33.126523 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:33.126523 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9" cmd=["/bin/sleep","15"]
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.126539    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" message="" err=<
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:33.126550    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" podUID="54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" containerName="main" containerID="cri-o://f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9"
Sep 17 23:29:33.126779 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.126643    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" podUID="54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" containerName="main" containerID="cri-o://f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9" gracePeriod=60
Sep 17 23:29:33.365386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.365363    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:33.415216 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-model-cache\") pod \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") "
Sep 17 23:29:33.415216 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-home\") pod \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") "
Sep 17 23:29:33.415373 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415221    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v8rg\" (UniqueName: \"kubernetes.io/projected/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-kube-api-access-6v8rg\") pod \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") "
Sep 17 23:29:33.415373 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415243    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-dshm\") pod \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") "
Sep 17 23:29:33.415373 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tls-certs\") pod \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") "
Sep 17 23:29:33.415373 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tmp-dir\") pod \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\" (UID: \"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa\") "
Sep 17 23:29:33.415570 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" (UID: "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:33.415570 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-home" (OuterVolumeSpecName: "home") pod "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" (UID: "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:33.415570 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415560    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:33.415746 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415601    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:33.415746 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.415680    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" (UID: "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:33.417551 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.417531    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-kube-api-access-6v8rg" (OuterVolumeSpecName: "kube-api-access-6v8rg") pod "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" (UID: "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa"). InnerVolumeSpecName "kube-api-access-6v8rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:29:33.417648 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.417592    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" (UID: "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:29:33.417648 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.417617    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-dshm" (OuterVolumeSpecName: "dshm") pod "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" (UID: "54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:33.516531 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.516509    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8rg\" (UniqueName: \"kubernetes.io/projected/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-kube-api-access-6v8rg\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:33.516531 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.516531    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:33.516682 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.516559    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:33.516682 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.516568    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:29:33.908699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.908667    2566 generic.go:356] "Generic (PLEG): container finished" podID="54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" containerID="f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9" exitCode=0
Sep 17 23:29:33.908847 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.908715    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"
Sep 17 23:29:33.908847 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.908757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" event={"ID":"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa","Type":"ContainerDied","Data":"f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9"}
Sep 17 23:29:33.908847 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.908814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m" event={"ID":"54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa","Type":"ContainerDied","Data":"0645ecb96ce82a6ec5a02caa57561c3d49af20ac3738e5155d83134aaf3adac2"}
Sep 17 23:29:33.908847 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.908836    2566 scope.go:117] "RemoveContainer" containerID="f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9"
Sep 17 23:29:33.917677 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.917621    2566 scope.go:117] "RemoveContainer" containerID="f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9"
Sep 17 23:29:33.917891 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:29:33.917871    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9\": container with ID starting with f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9 not found: ID does not exist" containerID="f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9"
Sep 17 23:29:33.917943 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.917899    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9"} err="failed to get container status \"f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9\": rpc error: code = NotFound desc = could not find container \"f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9\": container with ID starting with f4dd1512fa7f7dacc154c4d5d254217e7e219c92008514b4cfb9454c86fec7e9 not found: ID does not exist"
Sep 17 23:29:33.929088 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.929068    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"]
Sep 17 23:29:33.932952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:33.932929    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvgq7m"]
Sep 17 23:29:34.013860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:34.013837    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" path="/var/lib/kubelet/pods/54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa/volumes"
Sep 17 23:29:35.648961 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:35.648925    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:29:45.649145 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:45.649107    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:29:47.052282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:47.052202    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main" containerID="cri-o://00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a" gracePeriod=45
Sep 17 23:29:55.649148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:55.649109    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:29:55.649524 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:29:55.649221    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:30:05.649004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:05.648919    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:30:15.649344 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:15.649297    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:30:17.309937 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:17.309872    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:30:17.446804 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:17.446775    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620818af-b6f2-438d-bb10-c6783655c662-tls-certs\") pod \"620818af-b6f2-438d-bb10-c6783655c662\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") "
Sep 17 23:30:17.446804 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:17.446809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx22d\" (UniqueName: \"kubernetes.io/projected/620818af-b6f2-438d-bb10-c6783655c662-kube-api-access-cx22d\") pod \"620818af-b6f2-438d-bb10-c6783655c662\" (UID: \"620818af-b6f2-438d-bb10-c6783655c662\") "
Sep 17 23:30:17.449050 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:17.449022    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620818af-b6f2-438d-bb10-c6783655c662-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "620818af-b6f2-438d-bb10-c6783655c662" (UID: "620818af-b6f2-438d-bb10-c6783655c662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:30:17.449179 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:17.449062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620818af-b6f2-438d-bb10-c6783655c662-kube-api-access-cx22d" (OuterVolumeSpecName: "kube-api-access-cx22d") pod "620818af-b6f2-438d-bb10-c6783655c662" (UID: "620818af-b6f2-438d-bb10-c6783655c662"). InnerVolumeSpecName "kube-api-access-cx22d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:17.548344 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:17.548318    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620818af-b6f2-438d-bb10-c6783655c662-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:30:17.548344 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:17.548342    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx22d\" (UniqueName: \"kubernetes.io/projected/620818af-b6f2-438d-bb10-c6783655c662-kube-api-access-cx22d\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:30:18.126164 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.126130    2566 generic.go:356] "Generic (PLEG): container finished" podID="620818af-b6f2-438d-bb10-c6783655c662" containerID="00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a" exitCode=0
Sep 17 23:30:18.126292 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.126184    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"
Sep 17 23:30:18.126292 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.126186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" event={"ID":"620818af-b6f2-438d-bb10-c6783655c662","Type":"ContainerDied","Data":"00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a"}
Sep 17 23:30:18.126379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.126320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw" event={"ID":"620818af-b6f2-438d-bb10-c6783655c662","Type":"ContainerDied","Data":"47bc7da10fc3df4f16116524e31737baa1b67c5a385402ebf3e1dca6ff11ff44"}
Sep 17 23:30:18.126379 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.126362    2566 scope.go:117] "RemoveContainer" containerID="00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a"
Sep 17 23:30:18.135130 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.135110    2566 scope.go:117] "RemoveContainer" containerID="00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a"
Sep 17 23:30:18.135429 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:30:18.135404    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a\": container with ID starting with 00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a not found: ID does not exist" containerID="00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a"
Sep 17 23:30:18.135479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.135437    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a"} err="failed to get container status \"00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a\": rpc error: code = NotFound desc = could not find container \"00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a\": container with ID starting with 00689e0d945f084d1c47ec4a042ca00632e024eb4c0c3c99c22ad484389c422a not found: ID does not exist"
Sep 17 23:30:18.142918 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.142892    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"]
Sep 17 23:30:18.144393 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:18.144371    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75q6hlw"]
Sep 17 23:30:20.019918 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:20.019876    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620818af-b6f2-438d-bb10-c6783655c662" path="/var/lib/kubelet/pods/620818af-b6f2-438d-bb10-c6783655c662/volumes"
Sep 17 23:30:38.532046 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.532016    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"]
Sep 17 23:30:38.532513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.532354    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" containerName="main"
Sep 17 23:30:38.532513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.532366    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" containerName="main"
Sep 17 23:30:38.532513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.532385    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main"
Sep 17 23:30:38.532513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.532391    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main"
Sep 17 23:30:38.532513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.532455    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="620818af-b6f2-438d-bb10-c6783655c662" containerName="main"
Sep 17 23:30:38.532513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.532466    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bdb8a0-cacf-4fca-8ea7-e3ee8ce7e2aa" containerName="main"
Sep 17 23:30:38.537424 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.537397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.540743 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.540718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 23:30:38.540941 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.540917    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-pr6mw\""
Sep 17 23:30:38.541106 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.541083    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 23:30:38.541882 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.541704    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 23:30:38.542007 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.541923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-8frb9\""
Sep 17 23:30:38.549647 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.548945    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"]
Sep 17 23:30:38.698589 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.698554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.698772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.698608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.698772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.698669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4spkm\" (UniqueName: \"kubernetes.io/projected/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-kube-api-access-4spkm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.698772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.698740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.698901 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.698782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.698901 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.698802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.747247 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.747212    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"]
Sep 17 23:30:38.750425 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.750404    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:38.752937 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.752915    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-x6z7b\""
Sep 17 23:30:38.761021 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.760997    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"]
Sep 17 23:30:38.799486 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.799486 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.799486 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.799758 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.799758 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.799758 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4spkm\" (UniqueName: \"kubernetes.io/projected/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-kube-api-access-4spkm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.799962 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.800016 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.799963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.800053 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.800032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.801554 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.801537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.801902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.801884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.807974 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.807945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4spkm\" (UniqueName: \"kubernetes.io/projected/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-kube-api-access-4spkm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.852530 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.852507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:38.901224 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.901181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7038c624-8c89-47b7-9d46-bcd214b4aa4a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfgss\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:38.901339 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:38.901246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvgm\" (UniqueName: \"kubernetes.io/projected/7038c624-8c89-47b7-9d46-bcd214b4aa4a-kube-api-access-wcvgm\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfgss\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:39.002487 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.002457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7038c624-8c89-47b7-9d46-bcd214b4aa4a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfgss\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:39.002646 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.002495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvgm\" (UniqueName: \"kubernetes.io/projected/7038c624-8c89-47b7-9d46-bcd214b4aa4a-kube-api-access-wcvgm\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfgss\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:39.004785 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.004766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7038c624-8c89-47b7-9d46-bcd214b4aa4a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfgss\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:39.010094 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.010074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvgm\" (UniqueName: \"kubernetes.io/projected/7038c624-8c89-47b7-9d46-bcd214b4aa4a-kube-api-access-wcvgm\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfgss\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:39.062733 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.062673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:39.124112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.124081    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"]
Sep 17 23:30:39.125530 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:30:39.125496    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bd3afa_cedc_4f5b_8f2c_0e5a8acd8c89.slice/crio-ba81c9e441b748317645ec38d0ed295da295b8e5839af7356a0bad2042679745 WatchSource:0}: Error finding container ba81c9e441b748317645ec38d0ed295da295b8e5839af7356a0bad2042679745: Status 404 returned error can't find the container with id ba81c9e441b748317645ec38d0ed295da295b8e5839af7356a0bad2042679745
Sep 17 23:30:39.233249 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.233220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" event={"ID":"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89","Type":"ContainerStarted","Data":"ba81c9e441b748317645ec38d0ed295da295b8e5839af7356a0bad2042679745"}
Sep 17 23:30:39.335065 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:39.334886    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"]
Sep 17 23:30:39.337603 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:30:39.337565    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7038c624_8c89_47b7_9d46_bcd214b4aa4a.slice/crio-0f28808a7e1def12818848a2e67ecf647272bdf32c97ec0a8ee580ba472df8b3 WatchSource:0}: Error finding container 0f28808a7e1def12818848a2e67ecf647272bdf32c97ec0a8ee580ba472df8b3: Status 404 returned error can't find the container with id 0f28808a7e1def12818848a2e67ecf647272bdf32c97ec0a8ee580ba472df8b3
Sep 17 23:30:40.053676 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.053646    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:30:40.060078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.060054    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:30:40.062975 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.062955    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:30:40.069174 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.069156    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:30:40.238545 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.238518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" event={"ID":"7038c624-8c89-47b7-9d46-bcd214b4aa4a","Type":"ContainerStarted","Data":"559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217"}
Sep 17 23:30:40.238693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.238564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" event={"ID":"7038c624-8c89-47b7-9d46-bcd214b4aa4a","Type":"ContainerStarted","Data":"0f28808a7e1def12818848a2e67ecf647272bdf32c97ec0a8ee580ba472df8b3"}
Sep 17 23:30:40.238693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.238652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:30:40.255742 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:40.255702    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" podStartSLOduration=2.255689195 podStartE2EDuration="2.255689195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:40.254353801 +0000 UTC m=+1200.706185290" watchObservedRunningTime="2026-09-17 23:30:40.255689195 +0000 UTC m=+1200.707520682"
Sep 17 23:30:41.245277 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:41.245238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" event={"ID":"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89","Type":"ContainerStarted","Data":"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"}
Sep 17 23:30:41.245674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:41.245364    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:42.252004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:42.251967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" event={"ID":"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89","Type":"ContainerStarted","Data":"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"}
Sep 17 23:30:42.270668 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:42.270593    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" podStartSLOduration=2.718103283 podStartE2EDuration="4.270562863s" totalImagesPullingTime="1.55245958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:42.269099404 +0000 UTC m=+1202.720930892" watchObservedRunningTime="2026-09-17 23:30:42.270562863 +0000 UTC m=+1202.722394352"
Sep 17 23:30:42.853481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:42.853452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:48.853475 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:48.853428    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:48.866216 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:48.866188    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:49.301960 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:49.301927    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:30:50.310896 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:30:50.310865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:31:11.247389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:11.247353    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:31:14.859658 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:14.859619    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"]
Sep 17 23:31:14.868780 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:14.868750    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"]
Sep 17 23:31:14.878075 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:31:14.878041    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:31:14.878075 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:31:14.878075 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:31:14.878075 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c" cmd=["/bin/sleep","15"]
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:14.878095    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" message="" err=<
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:31:14.878112    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="main" containerID="cri-o://5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"
Sep 17 23:31:14.878371 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:14.878154    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="main" containerID="cri-o://5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c" gracePeriod=60
Sep 17 23:31:15.095040 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.095014    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="llm-d-routing-sidecar" containerID="cri-o://1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223" gracePeriod=60
Sep 17 23:31:15.341810 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.341787    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:31:15.383928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.383899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tmp-dir\") pod \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") "
Sep 17 23:31:15.384063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.383960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-model-cache\") pod \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") "
Sep 17 23:31:15.384063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.383983    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-home\") pod \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") "
Sep 17 23:31:15.384063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-dshm\") pod \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") "
Sep 17 23:31:15.384223 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384081    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4spkm\" (UniqueName: \"kubernetes.io/projected/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-kube-api-access-4spkm\") pod \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") "
Sep 17 23:31:15.384223 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tls-certs\") pod \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\" (UID: \"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89\") "
Sep 17 23:31:15.384223 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384143    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" (UID: "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:15.384354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" (UID: "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:15.384354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384302    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-home" (OuterVolumeSpecName: "home") pod "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" (UID: "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:15.384441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384400    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:31:15.384441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384416    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:31:15.384441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.384431    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:31:15.386306 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.386281    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-dshm" (OuterVolumeSpecName: "dshm") pod "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" (UID: "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:15.386423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.386344    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" (UID: "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:15.386423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.386377    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-kube-api-access-4spkm" (OuterVolumeSpecName: "kube-api-access-4spkm") pod "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" (UID: "c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89"). InnerVolumeSpecName "kube-api-access-4spkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:15.420967 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.420945    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerID="5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c" exitCode=0
Sep 17 23:31:15.420967 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.420965    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerID="1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223" exitCode=0
Sep 17 23:31:15.421111 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.420996    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"
Sep 17 23:31:15.421111 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.420996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" event={"ID":"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89","Type":"ContainerDied","Data":"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"}
Sep 17 23:31:15.421111 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.421050    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" event={"ID":"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89","Type":"ContainerDied","Data":"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"}
Sep 17 23:31:15.421111 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.421068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" event={"ID":"c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89","Type":"ContainerDied","Data":"ba81c9e441b748317645ec38d0ed295da295b8e5839af7356a0bad2042679745"}
Sep 17 23:31:15.421111 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.421088    2566 scope.go:117] "RemoveContainer" containerID="5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"
Sep 17 23:31:15.430248 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.430222    2566 scope.go:117] "RemoveContainer" containerID="1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"
Sep 17 23:31:15.437753 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.437734    2566 scope.go:117] "RemoveContainer" containerID="5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"
Sep 17 23:31:15.437995 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:31:15.437978    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c\": container with ID starting with 5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c not found: ID does not exist" containerID="5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"
Sep 17 23:31:15.438071 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.438007    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"} err="failed to get container status \"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c\": rpc error: code = NotFound desc = could not find container \"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c\": container with ID starting with 5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c not found: ID does not exist"
Sep 17 23:31:15.438071 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.438031    2566 scope.go:117] "RemoveContainer" containerID="1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"
Sep 17 23:31:15.438314 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:31:15.438296    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223\": container with ID starting with 1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223 not found: ID does not exist" containerID="1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"
Sep 17 23:31:15.438354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.438322    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"} err="failed to get container status \"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223\": rpc error: code = NotFound desc = could not find container \"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223\": container with ID starting with 1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223 not found: ID does not exist"
Sep 17 23:31:15.438354 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.438337    2566 scope.go:117] "RemoveContainer" containerID="5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"
Sep 17 23:31:15.438568 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.438548    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c"} err="failed to get container status \"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c\": rpc error: code = NotFound desc = could not find container \"5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c\": container with ID starting with 5c8e30538d353f02d5e20ee6703363a691aca39396a3cd00e816d6643c40375c not found: ID does not exist"
Sep 17 23:31:15.438650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.438569    2566 scope.go:117] "RemoveContainer" containerID="1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"
Sep 17 23:31:15.438868 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.438843    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223"} err="failed to get container status \"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223\": rpc error: code = NotFound desc = could not find container \"1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223\": container with ID starting with 1c3165a571fc559792c285b815920231fc02673221f5e44e37a3792a03b28223 not found: ID does not exist"
Sep 17 23:31:15.444409 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.444388    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"]
Sep 17 23:31:15.449067 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.449046    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps"]
Sep 17 23:31:15.485793 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.485774    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:31:15.485878 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.485793    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4spkm\" (UniqueName: \"kubernetes.io/projected/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-kube-api-access-4spkm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:31:15.485878 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:15.485808    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:31:16.013925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:16.013894    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" path="/var/lib/kubelet/pods/c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89/volumes"
Sep 17 23:31:16.292052 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:16.291963    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-q6lps" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 23:31:21.247228 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:21.247177    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:31:29.881436 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:29.881380    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main" containerID="cri-o://559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217" gracePeriod=45
Sep 17 23:31:31.247287 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:31.247243    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:31:41.247846 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:41.247800    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:31:41.248267 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:41.247928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:31:51.246816 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:31:51.246770    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:32:00.132754 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.132731    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:32:00.214698 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.214639    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvgm\" (UniqueName: \"kubernetes.io/projected/7038c624-8c89-47b7-9d46-bcd214b4aa4a-kube-api-access-wcvgm\") pod \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") "
Sep 17 23:32:00.214833 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.214700    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7038c624-8c89-47b7-9d46-bcd214b4aa4a-tls-certs\") pod \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\" (UID: \"7038c624-8c89-47b7-9d46-bcd214b4aa4a\") "
Sep 17 23:32:00.216663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.216634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7038c624-8c89-47b7-9d46-bcd214b4aa4a-kube-api-access-wcvgm" (OuterVolumeSpecName: "kube-api-access-wcvgm") pod "7038c624-8c89-47b7-9d46-bcd214b4aa4a" (UID: "7038c624-8c89-47b7-9d46-bcd214b4aa4a"). InnerVolumeSpecName "kube-api-access-wcvgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:00.216779 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.216766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7038c624-8c89-47b7-9d46-bcd214b4aa4a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7038c624-8c89-47b7-9d46-bcd214b4aa4a" (UID: "7038c624-8c89-47b7-9d46-bcd214b4aa4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:00.315169 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.315146    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvgm\" (UniqueName: \"kubernetes.io/projected/7038c624-8c89-47b7-9d46-bcd214b4aa4a-kube-api-access-wcvgm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:32:00.315271 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.315170    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7038c624-8c89-47b7-9d46-bcd214b4aa4a-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:32:00.648470 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.648441    2566 generic.go:356] "Generic (PLEG): container finished" podID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerID="559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217" exitCode=0
Sep 17 23:32:00.648628 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.648484    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"
Sep 17 23:32:00.648628 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.648503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" event={"ID":"7038c624-8c89-47b7-9d46-bcd214b4aa4a","Type":"ContainerDied","Data":"559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217"}
Sep 17 23:32:00.648628 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.648546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss" event={"ID":"7038c624-8c89-47b7-9d46-bcd214b4aa4a","Type":"ContainerDied","Data":"0f28808a7e1def12818848a2e67ecf647272bdf32c97ec0a8ee580ba472df8b3"}
Sep 17 23:32:00.648628 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.648567    2566 scope.go:117] "RemoveContainer" containerID="559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217"
Sep 17 23:32:00.658053 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.658036    2566 scope.go:117] "RemoveContainer" containerID="559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217"
Sep 17 23:32:00.658303 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:32:00.658286    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217\": container with ID starting with 559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217 not found: ID does not exist" containerID="559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217"
Sep 17 23:32:00.658348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.658309    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217"} err="failed to get container status \"559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217\": rpc error: code = NotFound desc = could not find container \"559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217\": container with ID starting with 559b771a7e0a82d9a80b28a5fd81e95f0e9649e61b8a28831f3682f2f928f217 not found: ID does not exist"
Sep 17 23:32:00.669470 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.669447    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"]
Sep 17 23:32:00.675272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:00.675249    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfgss"]
Sep 17 23:32:02.014047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:02.014009    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" path="/var/lib/kubelet/pods/7038c624-8c89-47b7-9d46-bcd214b4aa4a/volumes"
Sep 17 23:32:16.323619 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.323586    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"]
Sep 17 23:32:16.324376 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.324350    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="llm-d-routing-sidecar"
Sep 17 23:32:16.324497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.324380    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="llm-d-routing-sidecar"
Sep 17 23:32:16.324497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.324411    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="main"
Sep 17 23:32:16.324497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.324420    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="main"
Sep 17 23:32:16.324497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.324461    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main"
Sep 17 23:32:16.324497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.324469    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main"
Sep 17 23:32:16.330150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.330119    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="llm-d-routing-sidecar"
Sep 17 23:32:16.330256 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.330247    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7038c624-8c89-47b7-9d46-bcd214b4aa4a" containerName="main"
Sep 17 23:32:16.330296 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.330259    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6bd3afa-cedc-4f5b-8f2c-0e5a8acd8c89" containerName="main"
Sep 17 23:32:16.333602 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.333567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.336804 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.336787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 23:32:16.336916 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.336870    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 23:32:16.336980 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.336970    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:32:16.338177 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.338156    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 23:32:16.338177 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.338171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-rbm66\""
Sep 17 23:32:16.338343 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.338171    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 23:32:16.343766 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.343746    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"]
Sep 17 23:32:16.433310 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.433423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-home\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.433423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-dshm\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.433423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.433423 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/646039cc-14ad-49db-a17b-835ec2758904-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.433563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nphn\" (UniqueName: \"kubernetes.io/projected/646039cc-14ad-49db-a17b-835ec2758904-kube-api-access-5nphn\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.433563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-model-cache\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.433563 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.433528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646039cc-14ad-49db-a17b-835ec2758904-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.534556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-model-cache\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.534693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646039cc-14ad-49db-a17b-835ec2758904-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.534693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.534693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-home\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.534693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-dshm\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.534693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.534693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/646039cc-14ad-49db-a17b-835ec2758904-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.535077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.534719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nphn\" (UniqueName: \"kubernetes.io/projected/646039cc-14ad-49db-a17b-835ec2758904-kube-api-access-5nphn\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.535077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.535000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-home\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.535077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.535017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-model-cache\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.535077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.535073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.535261 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.535145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.535403 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.535385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/646039cc-14ad-49db-a17b-835ec2758904-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.536969 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.536944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-dshm\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.537390 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.537369    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646039cc-14ad-49db-a17b-835ec2758904-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.542467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.542447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nphn\" (UniqueName: \"kubernetes.io/projected/646039cc-14ad-49db-a17b-835ec2758904-kube-api-access-5nphn\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-v77l2\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.572608 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.572587    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"]
Sep 17 23:32:16.577373 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.577328    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.587693 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.587669    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"]
Sep 17 23:32:16.645120 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.645098    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:16.736407 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.736374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.736543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.736413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2695e75c-05aa-4242-b065-262df50976c7-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.736543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.736450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.736543 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.736494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2695e75c-05aa-4242-b065-262df50976c7-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.837771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.837705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2695e75c-05aa-4242-b065-262df50976c7-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.837771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.837750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.837915 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.837783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2695e75c-05aa-4242-b065-262df50976c7-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.837915 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.837836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.838234 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.838210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.838302 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.838219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.838528 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.838509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2695e75c-05aa-4242-b065-262df50976c7-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.840137 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.840118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2695e75c-05aa-4242-b065-262df50976c7-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.889282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.889263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:32:16.917229 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.917204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"]
Sep 17 23:32:16.920308 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:32:16.920285    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646039cc_14ad_49db_a17b_835ec2758904.slice/crio-c62d87b0b0b10ec31d2d6dde2e35d45f60bdd112c6e317f500a3a1a2cfbe9202 WatchSource:0}: Error finding container c62d87b0b0b10ec31d2d6dde2e35d45f60bdd112c6e317f500a3a1a2cfbe9202: Status 404 returned error can't find the container with id c62d87b0b0b10ec31d2d6dde2e35d45f60bdd112c6e317f500a3a1a2cfbe9202
Sep 17 23:32:16.922154 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:16.922138    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:32:17.163271 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:17.163197    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"]
Sep 17 23:32:17.165394 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:32:17.165361    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2695e75c_05aa_4242_b065_262df50976c7.slice/crio-079558e324fba8a5d3a5d51eff8c83db11740269590d6d4b6a1dd8ce226d54bd WatchSource:0}: Error finding container 079558e324fba8a5d3a5d51eff8c83db11740269590d6d4b6a1dd8ce226d54bd: Status 404 returned error can't find the container with id 079558e324fba8a5d3a5d51eff8c83db11740269590d6d4b6a1dd8ce226d54bd
Sep 17 23:32:17.745078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:17.745029    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" event={"ID":"646039cc-14ad-49db-a17b-835ec2758904","Type":"ContainerStarted","Data":"c62d87b0b0b10ec31d2d6dde2e35d45f60bdd112c6e317f500a3a1a2cfbe9202"}
Sep 17 23:32:17.748024 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:17.747288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" event={"ID":"2695e75c-05aa-4242-b065-262df50976c7","Type":"ContainerStarted","Data":"079558e324fba8a5d3a5d51eff8c83db11740269590d6d4b6a1dd8ce226d54bd"}
Sep 17 23:32:21.782851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:21.782812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" event={"ID":"2695e75c-05aa-4242-b065-262df50976c7","Type":"ContainerStarted","Data":"ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826"}
Sep 17 23:32:21.784556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:21.784526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" event={"ID":"646039cc-14ad-49db-a17b-835ec2758904","Type":"ContainerStarted","Data":"5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264"}
Sep 17 23:32:23.799432 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:23.799398    2566 generic.go:356] "Generic (PLEG): container finished" podID="646039cc-14ad-49db-a17b-835ec2758904" containerID="5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264" exitCode=0
Sep 17 23:32:23.799852 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:23.799473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" event={"ID":"646039cc-14ad-49db-a17b-835ec2758904","Type":"ContainerDied","Data":"5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264"}
Sep 17 23:32:23.801045 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:23.801022    2566 generic.go:356] "Generic (PLEG): container finished" podID="2695e75c-05aa-4242-b065-262df50976c7" containerID="ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826" exitCode=0
Sep 17 23:32:23.801139 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:23.801087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" event={"ID":"2695e75c-05aa-4242-b065-262df50976c7","Type":"ContainerDied","Data":"ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826"}
Sep 17 23:32:24.810254 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:24.810215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" event={"ID":"646039cc-14ad-49db-a17b-835ec2758904","Type":"ContainerStarted","Data":"a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280"}
Sep 17 23:32:24.831700 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:24.831644    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" podStartSLOduration=2.396210145 podStartE2EDuration="8.831620835s" totalImagesPullingTime="3.959963737s" totalInitContainerRuntime="2.475446953s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:24.829607644 +0000 UTC m=+1305.281439135" watchObservedRunningTime="2026-09-17 23:32:24.831620835 +0000 UTC m=+1305.283452323"
Sep 17 23:32:25.645906 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:25.645863    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:26.645820 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:26.645686    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:26.663984 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:26.663956    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:26.839466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:26.839433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:32:40.109441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:32:40.109408    2566 scope.go:117] "RemoveContainer" containerID="3754f31b4b9a8a6c27eb0abf002f734e0e187911dd35503c174c32dc234e4c31"
Sep 17 23:33:11.083038 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:11.082998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" event={"ID":"2695e75c-05aa-4242-b065-262df50976c7","Type":"ContainerStarted","Data":"6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab"}
Sep 17 23:33:11.083667 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:11.083221    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:33:11.101752 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:11.101701    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" podStartSLOduration=2.588802235 podStartE2EDuration="55.101685315s" totalImagesPullingTime="49.990292562s" totalInitContainerRuntime="2.522590518s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:11.100008145 +0000 UTC m=+1351.551839636" watchObservedRunningTime="2026-09-17 23:33:11.101685315 +0000 UTC m=+1351.553516802"
Sep 17 23:33:42.098705 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:42.098664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:33:44.845798 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:44.845766    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7_2695e75c-05aa-4242-b065-262df50976c7/main/0.log"
Sep 17 23:33:45.178272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.178172    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"]
Sep 17 23:33:45.178472 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.178445    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" podUID="2695e75c-05aa-4242-b065-262df50976c7" containerName="main" containerID="cri-o://6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab" gracePeriod=30
Sep 17 23:33:45.180343 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.180314    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"]
Sep 17 23:33:45.190062 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:33:45.190030    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:33:45.190062 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:33:45.190062 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:33:45.190062 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280" cmd=["/bin/sleep","15"]
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.190079    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" message="" err=<
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:33:45.190092    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" podUID="646039cc-14ad-49db-a17b-835ec2758904" containerName="main" containerID="cri-o://a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280"
Sep 17 23:33:45.190257 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.190154    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" podUID="646039cc-14ad-49db-a17b-835ec2758904" containerName="main" containerID="cri-o://a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280" gracePeriod=60
Sep 17 23:33:45.462061 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.462034    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:33:45.533479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-dshm\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.533479 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-model-cache\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.533755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533629    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nphn\" (UniqueName: \"kubernetes.io/projected/646039cc-14ad-49db-a17b-835ec2758904-kube-api-access-5nphn\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.533755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533671    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-tmp-dir\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.533755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-kserve-provision-location\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.533755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533711    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-model-cache" (OuterVolumeSpecName: "model-cache") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.533755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533748    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646039cc-14ad-49db-a17b-835ec2758904-tls-certs\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.534029 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533799    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-home\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.534029 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533856    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/646039cc-14ad-49db-a17b-835ec2758904-cabundle-cert\") pod \"646039cc-14ad-49db-a17b-835ec2758904\" (UID: \"646039cc-14ad-49db-a17b-835ec2758904\") "
Sep 17 23:33:45.534029 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.533875    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.534190 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.534029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-home" (OuterVolumeSpecName: "home") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.534190 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.534143    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.534190 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.534161    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.534190 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.534173    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.534396 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.534327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646039cc-14ad-49db-a17b-835ec2758904-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:45.536102 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.536078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646039cc-14ad-49db-a17b-835ec2758904-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:45.536237 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.536212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646039cc-14ad-49db-a17b-835ec2758904-kube-api-access-5nphn" (OuterVolumeSpecName: "kube-api-access-5nphn") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "kube-api-access-5nphn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:45.536335 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.536321    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-dshm" (OuterVolumeSpecName: "dshm") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.566819 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.566788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "646039cc-14ad-49db-a17b-835ec2758904" (UID: "646039cc-14ad-49db-a17b-835ec2758904"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.635013 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.634976    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646039cc-14ad-49db-a17b-835ec2758904-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.635013 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.635014    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/646039cc-14ad-49db-a17b-835ec2758904-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.635013 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.635026    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.635243 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.635040    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nphn\" (UniqueName: \"kubernetes.io/projected/646039cc-14ad-49db-a17b-835ec2758904-kube-api-access-5nphn\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.635243 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.635053    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/646039cc-14ad-49db-a17b-835ec2758904-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.816290 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.816268    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:33:45.937830 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.937804    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2695e75c-05aa-4242-b065-262df50976c7-tls-certs\") pod \"2695e75c-05aa-4242-b065-262df50976c7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") "
Sep 17 23:33:45.938282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.937835    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2695e75c-05aa-4242-b065-262df50976c7-cabundle-cert\") pod \"2695e75c-05aa-4242-b065-262df50976c7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") "
Sep 17 23:33:45.938282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.937867    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-tmp-dir\") pod \"2695e75c-05aa-4242-b065-262df50976c7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") "
Sep 17 23:33:45.938282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.937902    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-kserve-provision-location\") pod \"2695e75c-05aa-4242-b065-262df50976c7\" (UID: \"2695e75c-05aa-4242-b065-262df50976c7\") "
Sep 17 23:33:45.938282 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.938246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2695e75c-05aa-4242-b065-262df50976c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2695e75c-05aa-4242-b065-262df50976c7" (UID: "2695e75c-05aa-4242-b065-262df50976c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:45.938439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.938330    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2695e75c-05aa-4242-b065-262df50976c7" (UID: "2695e75c-05aa-4242-b065-262df50976c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.940068 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.940044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2695e75c-05aa-4242-b065-262df50976c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2695e75c-05aa-4242-b065-262df50976c7" (UID: "2695e75c-05aa-4242-b065-262df50976c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:45.958027 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:45.957972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2695e75c-05aa-4242-b065-262df50976c7" (UID: "2695e75c-05aa-4242-b065-262df50976c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:46.038833 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.038810    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.038833 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.038832    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2695e75c-05aa-4242-b065-262df50976c7-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.038952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.038840    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2695e75c-05aa-4242-b065-262df50976c7-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.038952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.038851    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2695e75c-05aa-4242-b065-262df50976c7-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.256330 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.256295    2566 generic.go:356] "Generic (PLEG): container finished" podID="646039cc-14ad-49db-a17b-835ec2758904" containerID="a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280" exitCode=0
Sep 17 23:33:46.256493 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.256348    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"
Sep 17 23:33:46.256493 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.256342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" event={"ID":"646039cc-14ad-49db-a17b-835ec2758904","Type":"ContainerDied","Data":"a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280"}
Sep 17 23:33:46.256493 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.256473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2" event={"ID":"646039cc-14ad-49db-a17b-835ec2758904","Type":"ContainerDied","Data":"c62d87b0b0b10ec31d2d6dde2e35d45f60bdd112c6e317f500a3a1a2cfbe9202"}
Sep 17 23:33:46.256672 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.256505    2566 scope.go:117] "RemoveContainer" containerID="a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280"
Sep 17 23:33:46.258293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.258268    2566 generic.go:356] "Generic (PLEG): container finished" podID="2695e75c-05aa-4242-b065-262df50976c7" containerID="6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab" exitCode=0
Sep 17 23:33:46.258436 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.258337    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"
Sep 17 23:33:46.258436 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.258353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" event={"ID":"2695e75c-05aa-4242-b065-262df50976c7","Type":"ContainerDied","Data":"6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab"}
Sep 17 23:33:46.258436 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.258395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7" event={"ID":"2695e75c-05aa-4242-b065-262df50976c7","Type":"ContainerDied","Data":"079558e324fba8a5d3a5d51eff8c83db11740269590d6d4b6a1dd8ce226d54bd"}
Sep 17 23:33:46.265943 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.265922    2566 scope.go:117] "RemoveContainer" containerID="5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264"
Sep 17 23:33:46.273603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.273586    2566 scope.go:117] "RemoveContainer" containerID="a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280"
Sep 17 23:33:46.273863 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:33:46.273846    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280\": container with ID starting with a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280 not found: ID does not exist" containerID="a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280"
Sep 17 23:33:46.273943 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.273876    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280"} err="failed to get container status \"a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280\": rpc error: code = NotFound desc = could not find container \"a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280\": container with ID starting with a32f5bf1bcf890427c7b529ee374f740b032b36ad5453b49d68fc5fd49085280 not found: ID does not exist"
Sep 17 23:33:46.273943 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.273897    2566 scope.go:117] "RemoveContainer" containerID="5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264"
Sep 17 23:33:46.274144 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:33:46.274127    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264\": container with ID starting with 5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264 not found: ID does not exist" containerID="5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264"
Sep 17 23:33:46.274184 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.274150    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264"} err="failed to get container status \"5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264\": rpc error: code = NotFound desc = could not find container \"5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264\": container with ID starting with 5528836ad74eebbb419d1fa2ce56acb642c2240e1333b0ae7fa36c526d9ff264 not found: ID does not exist"
Sep 17 23:33:46.274184 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.274166    2566 scope.go:117] "RemoveContainer" containerID="6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab"
Sep 17 23:33:46.287645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.284827    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"]
Sep 17 23:33:46.287645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.286491    2566 scope.go:117] "RemoveContainer" containerID="ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826"
Sep 17 23:33:46.292020 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.291999    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-v77l2"]
Sep 17 23:33:46.294860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.294835    2566 scope.go:117] "RemoveContainer" containerID="6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab"
Sep 17 23:33:46.295123 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:33:46.295105    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab\": container with ID starting with 6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab not found: ID does not exist" containerID="6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab"
Sep 17 23:33:46.295200 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.295128    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab"} err="failed to get container status \"6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab\": rpc error: code = NotFound desc = could not find container \"6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab\": container with ID starting with 6eaa774ef13159f2be24e63b5bc5d50b180c227ba07c9ffed5ef6764cc29eeab not found: ID does not exist"
Sep 17 23:33:46.295200 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.295143    2566 scope.go:117] "RemoveContainer" containerID="ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826"
Sep 17 23:33:46.295415 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:33:46.295397    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826\": container with ID starting with ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826 not found: ID does not exist" containerID="ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826"
Sep 17 23:33:46.295474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.295420    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826"} err="failed to get container status \"ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826\": rpc error: code = NotFound desc = could not find container \"ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826\": container with ID starting with ec6fa4e2e794561dbdbc67c613685b4697ae7cd71fa70d2871201de3644bc826 not found: ID does not exist"
Sep 17 23:33:46.301643 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.301618    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"]
Sep 17 23:33:46.303228 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:46.303204    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-78b578b99b-lgdc7"]
Sep 17 23:33:48.014464 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:48.014420    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2695e75c-05aa-4242-b065-262df50976c7" path="/var/lib/kubelet/pods/2695e75c-05aa-4242-b065-262df50976c7/volumes"
Sep 17 23:33:48.014951 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:33:48.014931    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646039cc-14ad-49db-a17b-835ec2758904" path="/var/lib/kubelet/pods/646039cc-14ad-49db-a17b-835ec2758904/volumes"
Sep 17 23:34:41.083778 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.083677    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"]
Sep 17 23:34:41.084291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084208    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646039cc-14ad-49db-a17b-835ec2758904" containerName="storage-initializer"
Sep 17 23:34:41.084291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="646039cc-14ad-49db-a17b-835ec2758904" containerName="storage-initializer"
Sep 17 23:34:41.084291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084258    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2695e75c-05aa-4242-b065-262df50976c7" containerName="storage-initializer"
Sep 17 23:34:41.084291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084268    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2695e75c-05aa-4242-b065-262df50976c7" containerName="storage-initializer"
Sep 17 23:34:41.084291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084284    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646039cc-14ad-49db-a17b-835ec2758904" containerName="main"
Sep 17 23:34:41.084291 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084293    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="646039cc-14ad-49db-a17b-835ec2758904" containerName="main"
Sep 17 23:34:41.084511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084334    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2695e75c-05aa-4242-b065-262df50976c7" containerName="main"
Sep 17 23:34:41.084511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084343    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2695e75c-05aa-4242-b065-262df50976c7" containerName="main"
Sep 17 23:34:41.084511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084430    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="646039cc-14ad-49db-a17b-835ec2758904" containerName="main"
Sep 17 23:34:41.084511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.084444    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2695e75c-05aa-4242-b065-262df50976c7" containerName="main"
Sep 17 23:34:41.087189 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.087155    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.091172 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.091107    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 23:34:41.091172 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.091180    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 23:34:41.091469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.091328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-2qblg\""
Sep 17 23:34:41.091469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.091364    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 23:34:41.091469 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.091421    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:34:41.091799 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.091780    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 23:34:41.098900 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.098873    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"]
Sep 17 23:34:41.147265 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6jn\" (UniqueName: \"kubernetes.io/projected/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kube-api-access-bd6jn\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.147396 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.147396 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfaa0764-6ea4-4a05-aecd-1941de52e80b-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.147396 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-home\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.147570 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-dshm\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.147570 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.147570 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.147570 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.147553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248515 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfaa0764-6ea4-4a05-aecd-1941de52e80b-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-home\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-dshm\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6jn\" (UniqueName: \"kubernetes.io/projected/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kube-api-access-bd6jn\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.248999 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.248969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-home\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.249057 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.249023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.249157 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.249137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.249273 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.249253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.249353 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.249335    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfaa0764-6ea4-4a05-aecd-1941de52e80b-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.250901 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.250883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-dshm\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.250988 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.250970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.258270 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.258252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6jn\" (UniqueName: \"kubernetes.io/projected/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kube-api-access-bd6jn\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.305961 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.305931    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"]
Sep 17 23:34:41.308996 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.308980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.319972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.319952    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"]
Sep 17 23:34:41.350162 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.350107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933a6687-924c-47e4-8f09-fb52641e83a4-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.350162 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.350142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a6687-924c-47e4-8f09-fb52641e83a4-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.350298 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.350164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.350298 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.350282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.400860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.400834    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:41.451196 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.451170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933a6687-924c-47e4-8f09-fb52641e83a4-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.451313 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.451211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a6687-924c-47e4-8f09-fb52641e83a4-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.451426 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.451406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.451887 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.451513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.451887 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.451837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.451887 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.451880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.452092 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.452015    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933a6687-924c-47e4-8f09-fb52641e83a4-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.454365 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.454344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a6687-924c-47e4-8f09-fb52641e83a4-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.620347 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.620277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:41.669240 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.669078    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"]
Sep 17 23:34:41.672063 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:34:41.672030    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaa0764_6ea4_4a05_aecd_1941de52e80b.slice/crio-71ab5c199fe461c4a5a0c8f0a708e25060a951bcb6c3bb673f57c3822d7f8300 WatchSource:0}: Error finding container 71ab5c199fe461c4a5a0c8f0a708e25060a951bcb6c3bb673f57c3822d7f8300: Status 404 returned error can't find the container with id 71ab5c199fe461c4a5a0c8f0a708e25060a951bcb6c3bb673f57c3822d7f8300
Sep 17 23:34:41.895719 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:41.895697    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"]
Sep 17 23:34:41.897700 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:34:41.897677    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933a6687_924c_47e4_8f09_fb52641e83a4.slice/crio-d0cf2bf4710dd4afe370da4a58eaab8615867bc856c3666a4008fafc6dc24c64 WatchSource:0}: Error finding container d0cf2bf4710dd4afe370da4a58eaab8615867bc856c3666a4008fafc6dc24c64: Status 404 returned error can't find the container with id d0cf2bf4710dd4afe370da4a58eaab8615867bc856c3666a4008fafc6dc24c64
Sep 17 23:34:42.539619 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:42.539564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" event={"ID":"933a6687-924c-47e4-8f09-fb52641e83a4","Type":"ContainerStarted","Data":"4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f"}
Sep 17 23:34:42.540153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:42.539662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" event={"ID":"933a6687-924c-47e4-8f09-fb52641e83a4","Type":"ContainerStarted","Data":"d0cf2bf4710dd4afe370da4a58eaab8615867bc856c3666a4008fafc6dc24c64"}
Sep 17 23:34:42.541406 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:42.541373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" event={"ID":"cfaa0764-6ea4-4a05-aecd-1941de52e80b","Type":"ContainerStarted","Data":"32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f"}
Sep 17 23:34:42.541511 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:42.541432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" event={"ID":"cfaa0764-6ea4-4a05-aecd-1941de52e80b","Type":"ContainerStarted","Data":"71ab5c199fe461c4a5a0c8f0a708e25060a951bcb6c3bb673f57c3822d7f8300"}
Sep 17 23:34:44.554688 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:44.554649    2566 generic.go:356] "Generic (PLEG): container finished" podID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerID="32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f" exitCode=0
Sep 17 23:34:44.555166 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:44.554729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" event={"ID":"cfaa0764-6ea4-4a05-aecd-1941de52e80b","Type":"ContainerDied","Data":"32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f"}
Sep 17 23:34:45.560948 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:45.560914    2566 generic.go:356] "Generic (PLEG): container finished" podID="933a6687-924c-47e4-8f09-fb52641e83a4" containerID="4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f" exitCode=0
Sep 17 23:34:45.561391 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:45.560985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" event={"ID":"933a6687-924c-47e4-8f09-fb52641e83a4","Type":"ContainerDied","Data":"4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f"}
Sep 17 23:34:45.563051 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:45.563026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" event={"ID":"cfaa0764-6ea4-4a05-aecd-1941de52e80b","Type":"ContainerStarted","Data":"02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515"}
Sep 17 23:34:45.595967 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:45.595926    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" podStartSLOduration=1.9941749450000001 podStartE2EDuration="4.595910795s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.60173585s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:45.594077413 +0000 UTC m=+1446.045908933" watchObservedRunningTime="2026-09-17 23:34:45.595910795 +0000 UTC m=+1446.047742282"
Sep 17 23:34:46.401278 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:46.401244    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:46.570728 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:46.570690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" event={"ID":"933a6687-924c-47e4-8f09-fb52641e83a4","Type":"ContainerStarted","Data":"5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff"}
Sep 17 23:34:46.571116 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:46.571086    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:34:46.589873 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:46.589825    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" podStartSLOduration=2.9750792109999997 podStartE2EDuration="5.589811461s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.61473225s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:46.587902086 +0000 UTC m=+1447.039733575" watchObservedRunningTime="2026-09-17 23:34:46.589811461 +0000 UTC m=+1447.041643002"
Sep 17 23:34:51.401621 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:51.401560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:51.414909 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:51.414882    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:34:51.608093 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:34:51.608061    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:35:17.587721 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:17.587679    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:35:18.775301 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:18.775266    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8_933a6687-924c-47e4-8f09-fb52641e83a4/main/0.log"
Sep 17 23:35:18.940746 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:18.940703    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"]
Sep 17 23:35:18.941470 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:18.941429    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" podUID="933a6687-924c-47e4-8f09-fb52641e83a4" containerName="main" containerID="cri-o://5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff" gracePeriod=30
Sep 17 23:35:18.943294 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:18.943267    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"]
Sep 17 23:35:18.954694 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:35:18.954658    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:35:18.954694 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:35:18.954694 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:35:18.954694 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515" cmd=["/bin/sleep","15"]
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:18.954720    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" message="" err=<
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:35:18.954736    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerName="main" containerID="cri-o://02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515"
Sep 17 23:35:18.954968 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:18.954789    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerName="main" containerID="cri-o://02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515" gracePeriod=60
Sep 17 23:35:19.315760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.315698    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:35:19.463200 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463175    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-model-cache\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463335 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd6jn\" (UniqueName: \"kubernetes.io/projected/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kube-api-access-bd6jn\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463335 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfaa0764-6ea4-4a05-aecd-1941de52e80b-cabundle-cert\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463335 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tmp-dir\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463503 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kserve-provision-location\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463503 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-home\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463503 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463419    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-dshm\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463503 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463453    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.463503 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tls-certs\") pod \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\" (UID: \"cfaa0764-6ea4-4a05-aecd-1941de52e80b\") "
Sep 17 23:35:19.463787 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-home" (OuterVolumeSpecName: "home") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.463787 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaa0764-6ea4-4a05-aecd-1941de52e80b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:19.463927 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.463629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.464176 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.464118    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.464176 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.464141    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfaa0764-6ea4-4a05-aecd-1941de52e80b-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.464176 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.464154    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.464176 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.464168    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.465861 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.465834    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kube-api-access-bd6jn" (OuterVolumeSpecName: "kube-api-access-bd6jn") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "kube-api-access-bd6jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:35:19.466186 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.466160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-dshm" (OuterVolumeSpecName: "dshm") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.466362 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.466318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:19.485488 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.485455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfaa0764-6ea4-4a05-aecd-1941de52e80b" (UID: "cfaa0764-6ea4-4a05-aecd-1941de52e80b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.564757 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.564732    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd6jn\" (UniqueName: \"kubernetes.io/projected/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kube-api-access-bd6jn\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.564757 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.564757    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.564901 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.564769    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfaa0764-6ea4-4a05-aecd-1941de52e80b-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.564901 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.564777    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfaa0764-6ea4-4a05-aecd-1941de52e80b-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.584259 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.584236    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:35:19.740427 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.740397    2566 generic.go:356] "Generic (PLEG): container finished" podID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerID="02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515" exitCode=0
Sep 17 23:35:19.740585 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.740456    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"
Sep 17 23:35:19.740585 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.740490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" event={"ID":"cfaa0764-6ea4-4a05-aecd-1941de52e80b","Type":"ContainerDied","Data":"02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515"}
Sep 17 23:35:19.740708 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.740603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5" event={"ID":"cfaa0764-6ea4-4a05-aecd-1941de52e80b","Type":"ContainerDied","Data":"71ab5c199fe461c4a5a0c8f0a708e25060a951bcb6c3bb673f57c3822d7f8300"}
Sep 17 23:35:19.740708 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.740638    2566 scope.go:117] "RemoveContainer" containerID="02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515"
Sep 17 23:35:19.742507 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.742484    2566 generic.go:356] "Generic (PLEG): container finished" podID="933a6687-924c-47e4-8f09-fb52641e83a4" containerID="5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff" exitCode=0
Sep 17 23:35:19.742625 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.742527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" event={"ID":"933a6687-924c-47e4-8f09-fb52641e83a4","Type":"ContainerDied","Data":"5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff"}
Sep 17 23:35:19.742625 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.742553    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"
Sep 17 23:35:19.742625 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.742564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8" event={"ID":"933a6687-924c-47e4-8f09-fb52641e83a4","Type":"ContainerDied","Data":"d0cf2bf4710dd4afe370da4a58eaab8615867bc856c3666a4008fafc6dc24c64"}
Sep 17 23:35:19.750958 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.750940    2566 scope.go:117] "RemoveContainer" containerID="32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f"
Sep 17 23:35:19.758992 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.758976    2566 scope.go:117] "RemoveContainer" containerID="02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515"
Sep 17 23:35:19.759243 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:35:19.759226    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515\": container with ID starting with 02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515 not found: ID does not exist" containerID="02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515"
Sep 17 23:35:19.759319 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.759252    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515"} err="failed to get container status \"02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515\": rpc error: code = NotFound desc = could not find container \"02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515\": container with ID starting with 02704f58389fb40459bf797e978264f07cb2ff18e04cf59f9c2968f8d6b58515 not found: ID does not exist"
Sep 17 23:35:19.759319 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.759273    2566 scope.go:117] "RemoveContainer" containerID="32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f"
Sep 17 23:35:19.759511 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:35:19.759494    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f\": container with ID starting with 32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f not found: ID does not exist" containerID="32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f"
Sep 17 23:35:19.759603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.759518    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f"} err="failed to get container status \"32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f\": rpc error: code = NotFound desc = could not find container \"32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f\": container with ID starting with 32799bbd72750627cd609552de407c478240a10ae43eec58d77da14b98c8d97f not found: ID does not exist"
Sep 17 23:35:19.759603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.759537    2566 scope.go:117] "RemoveContainer" containerID="5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff"
Sep 17 23:35:19.763888 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.763859    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"]
Sep 17 23:35:19.766614 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.766589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a6687-924c-47e4-8f09-fb52641e83a4-tls-certs\") pod \"933a6687-924c-47e4-8f09-fb52641e83a4\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") "
Sep 17 23:35:19.766725 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.766688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-kserve-provision-location\") pod \"933a6687-924c-47e4-8f09-fb52641e83a4\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") "
Sep 17 23:35:19.766796 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.766726    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933a6687-924c-47e4-8f09-fb52641e83a4-cabundle-cert\") pod \"933a6687-924c-47e4-8f09-fb52641e83a4\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") "
Sep 17 23:35:19.766876 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.766808    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-tmp-dir\") pod \"933a6687-924c-47e4-8f09-fb52641e83a4\" (UID: \"933a6687-924c-47e4-8f09-fb52641e83a4\") "
Sep 17 23:35:19.767258 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.767235    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "933a6687-924c-47e4-8f09-fb52641e83a4" (UID: "933a6687-924c-47e4-8f09-fb52641e83a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.767336 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.767238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933a6687-924c-47e4-8f09-fb52641e83a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "933a6687-924c-47e4-8f09-fb52641e83a4" (UID: "933a6687-924c-47e4-8f09-fb52641e83a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:19.768196 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.768174    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-9jzr5"]
Sep 17 23:35:19.768622 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.768601    2566 scope.go:117] "RemoveContainer" containerID="4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f"
Sep 17 23:35:19.768976 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.768951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933a6687-924c-47e4-8f09-fb52641e83a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "933a6687-924c-47e4-8f09-fb52641e83a4" (UID: "933a6687-924c-47e4-8f09-fb52641e83a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:19.776103 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.776085    2566 scope.go:117] "RemoveContainer" containerID="5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff"
Sep 17 23:35:19.776430 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:35:19.776331    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff\": container with ID starting with 5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff not found: ID does not exist" containerID="5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff"
Sep 17 23:35:19.776430 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.776349    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff"} err="failed to get container status \"5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff\": rpc error: code = NotFound desc = could not find container \"5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff\": container with ID starting with 5ab969811e7c77e5dd3af97c89c57ff8baf37411ba037f18335d647c3fb1a7ff not found: ID does not exist"
Sep 17 23:35:19.776430 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.776364    2566 scope.go:117] "RemoveContainer" containerID="4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f"
Sep 17 23:35:19.776610 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:35:19.776567    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f\": container with ID starting with 4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f not found: ID does not exist" containerID="4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f"
Sep 17 23:35:19.776655 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.776613    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f"} err="failed to get container status \"4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f\": rpc error: code = NotFound desc = could not find container \"4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f\": container with ID starting with 4f8a77f9e2df81af1f3ef3eea8ab93a4d98fc71b11d3ba79b7b1f3c75b537d0f not found: ID does not exist"
Sep 17 23:35:19.788969 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.788949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "933a6687-924c-47e4-8f09-fb52641e83a4" (UID: "933a6687-924c-47e4-8f09-fb52641e83a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.868417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.868361    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.868417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.868383    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933a6687-924c-47e4-8f09-fb52641e83a4-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.868417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.868392    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a6687-924c-47e4-8f09-fb52641e83a4-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.868417 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:19.868400    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a6687-924c-47e4-8f09-fb52641e83a4-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:35:20.013826 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:20.013801    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" path="/var/lib/kubelet/pods/cfaa0764-6ea4-4a05-aecd-1941de52e80b/volumes"
Sep 17 23:35:20.059969 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:20.059944    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"]
Sep 17 23:35:20.063766 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:20.063743    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79685d7bd6-9r8l8"]
Sep 17 23:35:22.014324 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:22.014281    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="933a6687-924c-47e4-8f09-fb52641e83a4" path="/var/lib/kubelet/pods/933a6687-924c-47e4-8f09-fb52641e83a4/volumes"
Sep 17 23:35:40.094093 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:40.094065    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:35:40.097308 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:40.097288    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:35:40.101402 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:40.101376    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:35:40.104267 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:35:40.104246    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:36:23.312041 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312008    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"]
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312318    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933a6687-924c-47e4-8f09-fb52641e83a4" containerName="storage-initializer"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312329    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="933a6687-924c-47e4-8f09-fb52641e83a4" containerName="storage-initializer"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312349    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerName="storage-initializer"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312355    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerName="storage-initializer"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312363    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerName="main"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312368    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerName="main"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312381    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933a6687-924c-47e4-8f09-fb52641e83a4" containerName="main"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312386    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="933a6687-924c-47e4-8f09-fb52641e83a4" containerName="main"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312433    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="933a6687-924c-47e4-8f09-fb52641e83a4" containerName="main"
Sep 17 23:36:23.312467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.312445    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfaa0764-6ea4-4a05-aecd-1941de52e80b" containerName="main"
Sep 17 23:36:23.315362 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.315339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.318126 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.318100    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 23:36:23.318256 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.318139    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:36:23.318256 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.318168    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 23:36:23.318256 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.318177    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-cz6v9\""
Sep 17 23:36:23.319279 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.319254    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 23:36:23.319279 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.319272    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 23:36:23.326426 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.326404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"]
Sep 17 23:36:23.437854 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.437826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4504fc31-fba0-4732-88ed-6691ed3fe016-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.437983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.437864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.437983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.437892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-home\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.437983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.437914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-dshm\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.437983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.437952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-model-cache\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.437983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.437976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.438227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.438018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4504fc31-fba0-4732-88ed-6691ed3fe016-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.438227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.438088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnz6t\" (UniqueName: \"kubernetes.io/projected/4504fc31-fba0-4732-88ed-6691ed3fe016-kube-api-access-bnz6t\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.532466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.532437    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"]
Sep 17 23:36:23.535986 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.535964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.539394 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz6t\" (UniqueName: \"kubernetes.io/projected/4504fc31-fba0-4732-88ed-6691ed3fe016-kube-api-access-bnz6t\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4504fc31-fba0-4732-88ed-6691ed3fe016-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-home\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-dshm\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539756 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-model-cache\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539756 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539756 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4504fc31-fba0-4732-88ed-6691ed3fe016-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.539985 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.539961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-home\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.540289 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.540244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-model-cache\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.540480 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.540451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4504fc31-fba0-4732-88ed-6691ed3fe016-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.540960 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.540936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.541514 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.541488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.542539 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.542519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4504fc31-fba0-4732-88ed-6691ed3fe016-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.542704 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.542683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-dshm\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.545760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.545736    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"]
Sep 17 23:36:23.548649 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.548628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnz6t\" (UniqueName: \"kubernetes.io/projected/4504fc31-fba0-4732-88ed-6691ed3fe016-kube-api-access-bnz6t\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-wttq6\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.626318 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.626290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:23.640752 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.640726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.640855 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.640772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.640855 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.640813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d11f2-5255-4f21-9393-b2b1dd225d76-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.640855 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.640844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.742143 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.742107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.742293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.742150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.742293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.742194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d11f2-5255-4f21-9393-b2b1dd225d76-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.742293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.742218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.742729 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.742696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.742729 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.742724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.742882 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.742863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d11f2-5255-4f21-9393-b2b1dd225d76-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.744509 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.744482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.877686 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.877616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:23.904477 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:23.904449    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"]
Sep 17 23:36:23.906769 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:36:23.906728    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4504fc31_fba0_4732_88ed_6691ed3fe016.slice/crio-946add45128e570dffca7242278f11e8aaad9ff8d88d6fe576dd09983ff63669 WatchSource:0}: Error finding container 946add45128e570dffca7242278f11e8aaad9ff8d88d6fe576dd09983ff63669: Status 404 returned error can't find the container with id 946add45128e570dffca7242278f11e8aaad9ff8d88d6fe576dd09983ff63669
Sep 17 23:36:24.067687 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:24.067652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" event={"ID":"4504fc31-fba0-4732-88ed-6691ed3fe016","Type":"ContainerStarted","Data":"d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444"}
Sep 17 23:36:24.067871 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:24.067716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" event={"ID":"4504fc31-fba0-4732-88ed-6691ed3fe016","Type":"ContainerStarted","Data":"946add45128e570dffca7242278f11e8aaad9ff8d88d6fe576dd09983ff63669"}
Sep 17 23:36:24.168972 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:24.168941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"]
Sep 17 23:36:24.170564 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:36:24.170538    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2d11f2_5255_4f21_9393_b2b1dd225d76.slice/crio-b65c4e0c36899d015cc9c32286cb4ea3da1c4f01e340eeb1d6dff2fc0504950d WatchSource:0}: Error finding container b65c4e0c36899d015cc9c32286cb4ea3da1c4f01e340eeb1d6dff2fc0504950d: Status 404 returned error can't find the container with id b65c4e0c36899d015cc9c32286cb4ea3da1c4f01e340eeb1d6dff2fc0504950d
Sep 17 23:36:25.076605 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:25.076553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" event={"ID":"bb2d11f2-5255-4f21-9393-b2b1dd225d76","Type":"ContainerStarted","Data":"4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954"}
Sep 17 23:36:25.077018 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:25.076623    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" event={"ID":"bb2d11f2-5255-4f21-9393-b2b1dd225d76","Type":"ContainerStarted","Data":"b65c4e0c36899d015cc9c32286cb4ea3da1c4f01e340eeb1d6dff2fc0504950d"}
Sep 17 23:36:27.089426 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:27.089393    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerID="4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954" exitCode=0
Sep 17 23:36:27.089850 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:27.089465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" event={"ID":"bb2d11f2-5255-4f21-9393-b2b1dd225d76","Type":"ContainerDied","Data":"4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954"}
Sep 17 23:36:27.091335 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:27.091306    2566 generic.go:356] "Generic (PLEG): container finished" podID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerID="d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444" exitCode=0
Sep 17 23:36:27.091441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:27.091362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" event={"ID":"4504fc31-fba0-4732-88ed-6691ed3fe016","Type":"ContainerDied","Data":"d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444"}
Sep 17 23:36:28.098547 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:28.098504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" event={"ID":"bb2d11f2-5255-4f21-9393-b2b1dd225d76","Type":"ContainerStarted","Data":"56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934"}
Sep 17 23:36:28.099022 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:28.098773    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:36:28.100412 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:28.100392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" event={"ID":"4504fc31-fba0-4732-88ed-6691ed3fe016","Type":"ContainerStarted","Data":"c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2"}
Sep 17 23:36:28.118109 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:28.118064    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" podStartSLOduration=2.593949814 podStartE2EDuration="5.118050615s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.524100801s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:28.116320901 +0000 UTC m=+1548.568152390" watchObservedRunningTime="2026-09-17 23:36:28.118050615 +0000 UTC m=+1548.569882103"
Sep 17 23:36:28.135466 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:28.135418    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" podStartSLOduration=2.575251986 podStartE2EDuration="5.135400333s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.560148347s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:28.133043918 +0000 UTC m=+1548.584875422" watchObservedRunningTime="2026-09-17 23:36:28.135400333 +0000 UTC m=+1548.587231823"
Sep 17 23:36:28.626983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:28.626949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:33.627228 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:33.627187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:33.639681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:33.639655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:34.142784 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:34.142748    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:36:59.117851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:36:59.117817    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:37:02.128181 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:02.128136    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj_bb2d11f2-5255-4f21-9393-b2b1dd225d76/main/0.log"
Sep 17 23:37:22.259559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.259519    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"]
Sep 17 23:37:22.265120 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.265095    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"]
Sep 17 23:37:22.265388 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.265345    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" podUID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerName="main" containerID="cri-o://56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934" gracePeriod=30
Sep 17 23:37:22.270095 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:37:22.270059    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:37:22.270095 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:37:22.270095 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:37:22.270095 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2" cmd=["/bin/sleep","15"]
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.270121    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" message="" err=<
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:37:22.270138    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerName="main" containerID="cri-o://c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2"
Sep 17 23:37:22.270313 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.270178    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerName="main" containerID="cri-o://c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2" gracePeriod=60
Sep 17 23:37:22.625791 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.625758    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:37:22.719849 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.719820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4504fc31-fba0-4732-88ed-6691ed3fe016-tls-certs\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.719969 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.719861    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-home\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.720035 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.719963    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-kserve-provision-location\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.720035 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720009    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4504fc31-fba0-4732-88ed-6691ed3fe016-cabundle-cert\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.720140 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720060    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-tmp-dir\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.720140 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720126    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnz6t\" (UniqueName: \"kubernetes.io/projected/4504fc31-fba0-4732-88ed-6691ed3fe016-kube-api-access-bnz6t\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.720245 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720060    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-home" (OuterVolumeSpecName: "home") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.720245 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720183    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-model-cache\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.720245 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720231    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-dshm\") pod \"4504fc31-fba0-4732-88ed-6691ed3fe016\" (UID: \"4504fc31-fba0-4732-88ed-6691ed3fe016\") "
Sep 17 23:37:22.720388 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720339    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.720388 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4504fc31-fba0-4732-88ed-6691ed3fe016-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:22.720486 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720453    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-model-cache" (OuterVolumeSpecName: "model-cache") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.720618 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720569    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4504fc31-fba0-4732-88ed-6691ed3fe016-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.720726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720623    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.720726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720637    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.720726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.720647    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.722540 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.722514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4504fc31-fba0-4732-88ed-6691ed3fe016-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:22.722676 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.722621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4504fc31-fba0-4732-88ed-6691ed3fe016-kube-api-access-bnz6t" (OuterVolumeSpecName: "kube-api-access-bnz6t") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "kube-api-access-bnz6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:22.722751 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.722725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-dshm" (OuterVolumeSpecName: "dshm") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.741734 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.741708    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4504fc31-fba0-4732-88ed-6691ed3fe016" (UID: "4504fc31-fba0-4732-88ed-6691ed3fe016"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.821058 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.820995    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnz6t\" (UniqueName: \"kubernetes.io/projected/4504fc31-fba0-4732-88ed-6691ed3fe016-kube-api-access-bnz6t\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.821058 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.821020    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.821058 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.821029    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4504fc31-fba0-4732-88ed-6691ed3fe016-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.821058 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.821038    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4504fc31-fba0-4732-88ed-6691ed3fe016-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.894771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:22.894748    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:37:23.022448 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.022421    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d11f2-5255-4f21-9393-b2b1dd225d76-cabundle-cert\") pod \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") "
Sep 17 23:37:23.022638 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.022461    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-kserve-provision-location\") pod \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") "
Sep 17 23:37:23.022638 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.022490    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tls-certs\") pod \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") "
Sep 17 23:37:23.022638 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.022512    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tmp-dir\") pod \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\" (UID: \"bb2d11f2-5255-4f21-9393-b2b1dd225d76\") "
Sep 17 23:37:23.022860 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.022837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2d11f2-5255-4f21-9393-b2b1dd225d76-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb2d11f2-5255-4f21-9393-b2b1dd225d76" (UID: "bb2d11f2-5255-4f21-9393-b2b1dd225d76"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:23.023038 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.023014    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb2d11f2-5255-4f21-9393-b2b1dd225d76" (UID: "bb2d11f2-5255-4f21-9393-b2b1dd225d76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:23.024559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.024542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb2d11f2-5255-4f21-9393-b2b1dd225d76" (UID: "bb2d11f2-5255-4f21-9393-b2b1dd225d76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:23.045015 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.044987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb2d11f2-5255-4f21-9393-b2b1dd225d76" (UID: "bb2d11f2-5255-4f21-9393-b2b1dd225d76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:23.123863 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.123808    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d11f2-5255-4f21-9393-b2b1dd225d76-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:23.123863 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.123828    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:23.123863 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.123844    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:23.123863 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.123857    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2d11f2-5255-4f21-9393-b2b1dd225d76-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:37:23.378659 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.378565    2566 generic.go:356] "Generic (PLEG): container finished" podID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerID="c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2" exitCode=0
Sep 17 23:37:23.378659 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.378653    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"
Sep 17 23:37:23.379131 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.378651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" event={"ID":"4504fc31-fba0-4732-88ed-6691ed3fe016","Type":"ContainerDied","Data":"c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2"}
Sep 17 23:37:23.379131 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.378704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6" event={"ID":"4504fc31-fba0-4732-88ed-6691ed3fe016","Type":"ContainerDied","Data":"946add45128e570dffca7242278f11e8aaad9ff8d88d6fe576dd09983ff63669"}
Sep 17 23:37:23.379131 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.378727    2566 scope.go:117] "RemoveContainer" containerID="c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2"
Sep 17 23:37:23.381114 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.381084    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerID="56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934" exitCode=0
Sep 17 23:37:23.381235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.381148    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"
Sep 17 23:37:23.381235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.381171    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" event={"ID":"bb2d11f2-5255-4f21-9393-b2b1dd225d76","Type":"ContainerDied","Data":"56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934"}
Sep 17 23:37:23.381326 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.381235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj" event={"ID":"bb2d11f2-5255-4f21-9393-b2b1dd225d76","Type":"ContainerDied","Data":"b65c4e0c36899d015cc9c32286cb4ea3da1c4f01e340eeb1d6dff2fc0504950d"}
Sep 17 23:37:23.393949 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.393934    2566 scope.go:117] "RemoveContainer" containerID="d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444"
Sep 17 23:37:23.401514 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.401498    2566 scope.go:117] "RemoveContainer" containerID="c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2"
Sep 17 23:37:23.401804 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:37:23.401787    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2\": container with ID starting with c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2 not found: ID does not exist" containerID="c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2"
Sep 17 23:37:23.401857 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.401811    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2"} err="failed to get container status \"c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2\": rpc error: code = NotFound desc = could not find container \"c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2\": container with ID starting with c1d1c7f96644dcd15b2ea79a6e549e274a51132d7e755fdd7485cbdf997fe7c2 not found: ID does not exist"
Sep 17 23:37:23.401857 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.401827    2566 scope.go:117] "RemoveContainer" containerID="d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444"
Sep 17 23:37:23.402104 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:37:23.402086    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444\": container with ID starting with d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444 not found: ID does not exist" containerID="d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444"
Sep 17 23:37:23.402158 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.402109    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444"} err="failed to get container status \"d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444\": rpc error: code = NotFound desc = could not find container \"d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444\": container with ID starting with d966fc88ddb0fcd22dff280f8c2e80e720a6a142da4ebf41e3ae66f10da86444 not found: ID does not exist"
Sep 17 23:37:23.402158 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.402125    2566 scope.go:117] "RemoveContainer" containerID="56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934"
Sep 17 23:37:23.405150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.405124    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"]
Sep 17 23:37:23.409980 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.409959    2566 scope.go:117] "RemoveContainer" containerID="4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954"
Sep 17 23:37:23.411293 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.411271    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d4ccc6c7d-dtvcj"]
Sep 17 23:37:23.417593 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.417555    2566 scope.go:117] "RemoveContainer" containerID="56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934"
Sep 17 23:37:23.417915 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:37:23.417897    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934\": container with ID starting with 56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934 not found: ID does not exist" containerID="56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934"
Sep 17 23:37:23.418000 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.417924    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934"} err="failed to get container status \"56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934\": rpc error: code = NotFound desc = could not find container \"56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934\": container with ID starting with 56ae1d7178636560313fe07e1ebd208bd55c709ef2c4a19e4749d17a2a293934 not found: ID does not exist"
Sep 17 23:37:23.418000 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.417946    2566 scope.go:117] "RemoveContainer" containerID="4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954"
Sep 17 23:37:23.418205 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:37:23.418188    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954\": container with ID starting with 4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954 not found: ID does not exist" containerID="4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954"
Sep 17 23:37:23.418250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.418211    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954"} err="failed to get container status \"4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954\": rpc error: code = NotFound desc = could not find container \"4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954\": container with ID starting with 4c3896eed132e4ee23ba7575d65f8d909fab485d89870ee467cb1465690d4954 not found: ID does not exist"
Sep 17 23:37:23.422911 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.422889    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"]
Sep 17 23:37:23.428977 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:23.428957    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-wttq6"]
Sep 17 23:37:24.014535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:24.014498    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" path="/var/lib/kubelet/pods/4504fc31-fba0-4732-88ed-6691ed3fe016/volumes"
Sep 17 23:37:24.015083 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:37:24.015053    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" path="/var/lib/kubelet/pods/bb2d11f2-5255-4f21-9393-b2b1dd225d76/volumes"
Sep 17 23:38:21.598374 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598326    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"]
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598705    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerName="storage-initializer"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598717    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerName="storage-initializer"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598725    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerName="main"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598731    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerName="main"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598740    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerName="storage-initializer"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598745    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerName="storage-initializer"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598756    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerName="main"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598761    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerName="main"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598824    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2d11f2-5255-4f21-9393-b2b1dd225d76" containerName="main"
Sep 17 23:38:21.598851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.598832    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4504fc31-fba0-4732-88ed-6691ed3fe016" containerName="main"
Sep 17 23:38:21.601925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.601902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.605782 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.605759    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 23:38:21.605908 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.605823    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 23:38:21.605908 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.605840    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 23:38:21.605908 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.605879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-kzvgj\""
Sep 17 23:38:21.611441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.611416    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"]
Sep 17 23:38:21.651247 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.651221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsn5\" (UniqueName: \"kubernetes.io/projected/19990dcf-4fa3-4a71-b881-1588dda2a9af-kube-api-access-rmsn5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.651346 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.651255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19990dcf-4fa3-4a71-b881-1588dda2a9af-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.651346 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.651282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.651346 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.651318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.651346 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.651340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.651505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.651382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.752455 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.752428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsn5\" (UniqueName: \"kubernetes.io/projected/19990dcf-4fa3-4a71-b881-1588dda2a9af-kube-api-access-rmsn5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.752625 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.752463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19990dcf-4fa3-4a71-b881-1588dda2a9af-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.752625 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.752490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.752745 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.752620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.752745 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.752659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.752745 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.752685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.753042 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.753022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.753116 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.753102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.753180 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.753101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.754700 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.754678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.755038 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.755019    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19990dcf-4fa3-4a71-b881-1588dda2a9af-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.761861 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.761835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsn5\" (UniqueName: \"kubernetes.io/projected/19990dcf-4fa3-4a71-b881-1588dda2a9af-kube-api-access-rmsn5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.851754 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.851694    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"]
Sep 17 23:38:21.855219 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.855200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:21.857737 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.857715    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-wrsfb\""
Sep 17 23:38:21.863773 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.863753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"]
Sep 17 23:38:21.912797 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.912776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:21.953933 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.953901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztp4\" (UniqueName: \"kubernetes.io/projected/88df22e1-4fef-4cae-b6f5-ee769529cdef-kube-api-access-jztp4\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:21.954055 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:21.953952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88df22e1-4fef-4cae-b6f5-ee769529cdef-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:22.054402 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.054369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztp4\" (UniqueName: \"kubernetes.io/projected/88df22e1-4fef-4cae-b6f5-ee769529cdef-kube-api-access-jztp4\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:22.054560 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.054429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88df22e1-4fef-4cae-b6f5-ee769529cdef-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:22.056889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.056850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88df22e1-4fef-4cae-b6f5-ee769529cdef-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:22.062204 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.062176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztp4\" (UniqueName: \"kubernetes.io/projected/88df22e1-4fef-4cae-b6f5-ee769529cdef-kube-api-access-jztp4\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:22.166757 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.166679    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:22.183916 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.183880    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"]
Sep 17 23:38:22.185853 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:38:22.185831    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19990dcf_4fa3_4a71_b881_1588dda2a9af.slice/crio-973dd7c1ba6f2ea1528d76cc99fc472233d72261fa386aeceb7622729739bb20 WatchSource:0}: Error finding container 973dd7c1ba6f2ea1528d76cc99fc472233d72261fa386aeceb7622729739bb20: Status 404 returned error can't find the container with id 973dd7c1ba6f2ea1528d76cc99fc472233d72261fa386aeceb7622729739bb20
Sep 17 23:38:22.439657 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.439432    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"]
Sep 17 23:38:22.442073 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:38:22.442051    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88df22e1_4fef_4cae_b6f5_ee769529cdef.slice/crio-1ea7736ee64407e78911be4639853121373aec889d80ca6fa3adb6bc5077341a WatchSource:0}: Error finding container 1ea7736ee64407e78911be4639853121373aec889d80ca6fa3adb6bc5077341a: Status 404 returned error can't find the container with id 1ea7736ee64407e78911be4639853121373aec889d80ca6fa3adb6bc5077341a
Sep 17 23:38:22.683018 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.682980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" event={"ID":"88df22e1-4fef-4cae-b6f5-ee769529cdef","Type":"ContainerStarted","Data":"8c13b295451426bb13fd16dacd0f5ec47ac92a3062f6bc3412b759a276f21979"}
Sep 17 23:38:22.683467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.683054    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:22.683467 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.683077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" event={"ID":"88df22e1-4fef-4cae-b6f5-ee769529cdef","Type":"ContainerStarted","Data":"1ea7736ee64407e78911be4639853121373aec889d80ca6fa3adb6bc5077341a"}
Sep 17 23:38:22.684677 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.684649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" event={"ID":"19990dcf-4fa3-4a71-b881-1588dda2a9af","Type":"ContainerStarted","Data":"5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c"}
Sep 17 23:38:22.684787 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.684700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" event={"ID":"19990dcf-4fa3-4a71-b881-1588dda2a9af","Type":"ContainerStarted","Data":"973dd7c1ba6f2ea1528d76cc99fc472233d72261fa386aeceb7622729739bb20"}
Sep 17 23:38:22.702553 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.702476    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" podStartSLOduration=1.702461611 podStartE2EDuration="1.702461611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:22.701293379 +0000 UTC m=+1663.153124868" watchObservedRunningTime="2026-09-17 23:38:22.702461611 +0000 UTC m=+1663.154293098"
Sep 17 23:38:22.718018 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.717970    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" podStartSLOduration=1.717959527 podStartE2EDuration="1.717959527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:22.716820987 +0000 UTC m=+1663.168652475" watchObservedRunningTime="2026-09-17 23:38:22.717959527 +0000 UTC m=+1663.169791014"
Sep 17 23:38:22.913682 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:22.913647    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:31.913150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:31.913114    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:31.925401 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:31.925366    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:32.741218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:32.741187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:53.691349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:53.691308    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:38:54.838098 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:54.838010    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"]
Sep 17 23:38:54.852143 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:54.852112    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"]
Sep 17 23:38:54.865106 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:38:54.865072    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:38:54.865106 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:38:54.865106 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:38:54.865106 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c" cmd=["/bin/sleep","15"]
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:54.865123    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" message="" err=<
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:38:54.865136    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" podUID="19990dcf-4fa3-4a71-b881-1588dda2a9af" containerName="main" containerID="cri-o://5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c"
Sep 17 23:38:54.865451 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:54.865170    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" podUID="19990dcf-4fa3-4a71-b881-1588dda2a9af" containerName="main" containerID="cri-o://5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c" gracePeriod=60
Sep 17 23:38:55.609246 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.609221    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:55.713589 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713557    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmsn5\" (UniqueName: \"kubernetes.io/projected/19990dcf-4fa3-4a71-b881-1588dda2a9af-kube-api-access-rmsn5\") pod \"19990dcf-4fa3-4a71-b881-1588dda2a9af\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") "
Sep 17 23:38:55.713722 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713641    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19990dcf-4fa3-4a71-b881-1588dda2a9af-tls-certs\") pod \"19990dcf-4fa3-4a71-b881-1588dda2a9af\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") "
Sep 17 23:38:55.713722 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713662    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-dshm\") pod \"19990dcf-4fa3-4a71-b881-1588dda2a9af\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") "
Sep 17 23:38:55.713722 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713684    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-model-cache\") pod \"19990dcf-4fa3-4a71-b881-1588dda2a9af\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") "
Sep 17 23:38:55.713893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-home\") pod \"19990dcf-4fa3-4a71-b881-1588dda2a9af\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") "
Sep 17 23:38:55.713893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713758    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-tmp-dir\") pod \"19990dcf-4fa3-4a71-b881-1588dda2a9af\" (UID: \"19990dcf-4fa3-4a71-b881-1588dda2a9af\") "
Sep 17 23:38:55.713996 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713969    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-home" (OuterVolumeSpecName: "home") pod "19990dcf-4fa3-4a71-b881-1588dda2a9af" (UID: "19990dcf-4fa3-4a71-b881-1588dda2a9af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:55.714050 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.713988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-model-cache" (OuterVolumeSpecName: "model-cache") pod "19990dcf-4fa3-4a71-b881-1588dda2a9af" (UID: "19990dcf-4fa3-4a71-b881-1588dda2a9af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:55.714134 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.714115    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19990dcf-4fa3-4a71-b881-1588dda2a9af" (UID: "19990dcf-4fa3-4a71-b881-1588dda2a9af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:55.715906 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.715880    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19990dcf-4fa3-4a71-b881-1588dda2a9af-kube-api-access-rmsn5" (OuterVolumeSpecName: "kube-api-access-rmsn5") pod "19990dcf-4fa3-4a71-b881-1588dda2a9af" (UID: "19990dcf-4fa3-4a71-b881-1588dda2a9af"). InnerVolumeSpecName "kube-api-access-rmsn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:55.716003 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.715901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-dshm" (OuterVolumeSpecName: "dshm") pod "19990dcf-4fa3-4a71-b881-1588dda2a9af" (UID: "19990dcf-4fa3-4a71-b881-1588dda2a9af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:55.716003 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.715886    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19990dcf-4fa3-4a71-b881-1588dda2a9af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19990dcf-4fa3-4a71-b881-1588dda2a9af" (UID: "19990dcf-4fa3-4a71-b881-1588dda2a9af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:38:55.814699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.814674    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:38:55.814699 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.814698    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmsn5\" (UniqueName: \"kubernetes.io/projected/19990dcf-4fa3-4a71-b881-1588dda2a9af-kube-api-access-rmsn5\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:38:55.814849 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.814709    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19990dcf-4fa3-4a71-b881-1588dda2a9af-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:38:55.814849 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.814717    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:38:55.814849 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.814726    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:38:55.814849 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.814734    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19990dcf-4fa3-4a71-b881-1588dda2a9af-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:38:55.851193 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.851167    2566 generic.go:356] "Generic (PLEG): container finished" podID="19990dcf-4fa3-4a71-b881-1588dda2a9af" containerID="5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c" exitCode=0
Sep 17 23:38:55.851542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.851207    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"
Sep 17 23:38:55.851542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.851212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" event={"ID":"19990dcf-4fa3-4a71-b881-1588dda2a9af","Type":"ContainerDied","Data":"5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c"}
Sep 17 23:38:55.851542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.851254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5" event={"ID":"19990dcf-4fa3-4a71-b881-1588dda2a9af","Type":"ContainerDied","Data":"973dd7c1ba6f2ea1528d76cc99fc472233d72261fa386aeceb7622729739bb20"}
Sep 17 23:38:55.851542 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.851275    2566 scope.go:117] "RemoveContainer" containerID="5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c"
Sep 17 23:38:55.863348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.863327    2566 scope.go:117] "RemoveContainer" containerID="5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c"
Sep 17 23:38:55.863789 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:38:55.863767    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c\": container with ID starting with 5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c not found: ID does not exist" containerID="5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c"
Sep 17 23:38:55.863871 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.863795    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c"} err="failed to get container status \"5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c\": rpc error: code = NotFound desc = could not find container \"5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c\": container with ID starting with 5a1bb75f5f57d5f26af062b141c62ba8e0be4fa482281c976cfd584840c10e2c not found: ID does not exist"
Sep 17 23:38:55.885924 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.885900    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"]
Sep 17 23:38:55.896164 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:55.896140    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-749f5"]
Sep 17 23:38:56.014923 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:38:56.014898    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19990dcf-4fa3-4a71-b881-1588dda2a9af" path="/var/lib/kubelet/pods/19990dcf-4fa3-4a71-b881-1588dda2a9af/volumes"
Sep 17 23:39:03.691327 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:03.691243    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:39:09.866571 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:09.866517    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main" containerID="cri-o://8c13b295451426bb13fd16dacd0f5ec47ac92a3062f6bc3412b759a276f21979" gracePeriod=45
Sep 17 23:39:13.691490 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:13.691450    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:39:23.691566 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:23.691526    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:39:23.692035 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:23.691695    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:39:33.691502 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:33.691456    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:39:40.066121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.066087    2566 generic.go:356] "Generic (PLEG): container finished" podID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerID="8c13b295451426bb13fd16dacd0f5ec47ac92a3062f6bc3412b759a276f21979" exitCode=0
Sep 17 23:39:40.066455 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.066163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" event={"ID":"88df22e1-4fef-4cae-b6f5-ee769529cdef","Type":"ContainerDied","Data":"8c13b295451426bb13fd16dacd0f5ec47ac92a3062f6bc3412b759a276f21979"}
Sep 17 23:39:40.114386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.114368    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:39:40.131981 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.131922    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jztp4\" (UniqueName: \"kubernetes.io/projected/88df22e1-4fef-4cae-b6f5-ee769529cdef-kube-api-access-jztp4\") pod \"88df22e1-4fef-4cae-b6f5-ee769529cdef\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") "
Sep 17 23:39:40.132663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.132339    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88df22e1-4fef-4cae-b6f5-ee769529cdef-tls-certs\") pod \"88df22e1-4fef-4cae-b6f5-ee769529cdef\" (UID: \"88df22e1-4fef-4cae-b6f5-ee769529cdef\") "
Sep 17 23:39:40.134483 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.134443    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88df22e1-4fef-4cae-b6f5-ee769529cdef-kube-api-access-jztp4" (OuterVolumeSpecName: "kube-api-access-jztp4") pod "88df22e1-4fef-4cae-b6f5-ee769529cdef" (UID: "88df22e1-4fef-4cae-b6f5-ee769529cdef"). InnerVolumeSpecName "kube-api-access-jztp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:40.134854 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.134834    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88df22e1-4fef-4cae-b6f5-ee769529cdef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88df22e1-4fef-4cae-b6f5-ee769529cdef" (UID: "88df22e1-4fef-4cae-b6f5-ee769529cdef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:39:40.233902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.233876    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jztp4\" (UniqueName: \"kubernetes.io/projected/88df22e1-4fef-4cae-b6f5-ee769529cdef-kube-api-access-jztp4\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:39:40.233902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:40.233902    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88df22e1-4fef-4cae-b6f5-ee769529cdef-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:39:41.072705 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:41.072673    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"
Sep 17 23:39:41.073170 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:41.072707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq" event={"ID":"88df22e1-4fef-4cae-b6f5-ee769529cdef","Type":"ContainerDied","Data":"1ea7736ee64407e78911be4639853121373aec889d80ca6fa3adb6bc5077341a"}
Sep 17 23:39:41.073170 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:41.072778    2566 scope.go:117] "RemoveContainer" containerID="8c13b295451426bb13fd16dacd0f5ec47ac92a3062f6bc3412b759a276f21979"
Sep 17 23:39:41.094660 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:41.094559    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"]
Sep 17 23:39:41.097874 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:41.097849    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schej6dzq"]
Sep 17 23:39:42.014112 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:42.014080    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" path="/var/lib/kubelet/pods/88df22e1-4fef-4cae-b6f5-ee769529cdef/volumes"
Sep 17 23:39:54.075680 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.075628    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"]
Sep 17 23:39:54.076085 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.076018    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19990dcf-4fa3-4a71-b881-1588dda2a9af" containerName="main"
Sep 17 23:39:54.076085 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.076031    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19990dcf-4fa3-4a71-b881-1588dda2a9af" containerName="main"
Sep 17 23:39:54.076085 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.076063    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main"
Sep 17 23:39:54.076085 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.076069    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main"
Sep 17 23:39:54.076225 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.076141    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="88df22e1-4fef-4cae-b6f5-ee769529cdef" containerName="main"
Sep 17 23:39:54.076225 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.076151    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="19990dcf-4fa3-4a71-b881-1588dda2a9af" containerName="main"
Sep 17 23:39:54.078763 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.078741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.082381 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.082356    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 23:39:54.082503 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.082407    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 23:39:54.082503 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.082477    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 23:39:54.082674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.082550    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-nb7xx\""
Sep 17 23:39:54.088867 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.088837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"]
Sep 17 23:39:54.136129 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.136107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.136227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.136131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.136227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.136149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833ca489-0771-4d2a-bc26-acf4cf476b1b-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.136227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.136168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngqm\" (UniqueName: \"kubernetes.io/projected/833ca489-0771-4d2a-bc26-acf4cf476b1b-kube-api-access-wngqm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.136227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.136223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.136380 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.136251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.236548 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.236526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.236664 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.236557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.236664 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.236626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.236664 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.236644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.236664 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.236661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833ca489-0771-4d2a-bc26-acf4cf476b1b-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.236871 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.236683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngqm\" (UniqueName: \"kubernetes.io/projected/833ca489-0771-4d2a-bc26-acf4cf476b1b-kube-api-access-wngqm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.236975 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.236954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.237036 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.237006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.237075 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.237052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.238898 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.238878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.239093 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.239077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833ca489-0771-4d2a-bc26-acf4cf476b1b-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.245258 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.245235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngqm\" (UniqueName: \"kubernetes.io/projected/833ca489-0771-4d2a-bc26-acf4cf476b1b-kube-api-access-wngqm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.390063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.389998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:39:54.659494 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:54.659472    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"]
Sep 17 23:39:54.661678 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:39:54.661646    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833ca489_0771_4d2a_bc26_acf4cf476b1b.slice/crio-7b37ce661723a9944a5e48d0b8c64df2c09dd33c6e9081098719b3115b418db5 WatchSource:0}: Error finding container 7b37ce661723a9944a5e48d0b8c64df2c09dd33c6e9081098719b3115b418db5: Status 404 returned error can't find the container with id 7b37ce661723a9944a5e48d0b8c64df2c09dd33c6e9081098719b3115b418db5
Sep 17 23:39:55.140341 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:55.140295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" event={"ID":"833ca489-0771-4d2a-bc26-acf4cf476b1b","Type":"ContainerStarted","Data":"074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c"}
Sep 17 23:39:55.140841 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:55.140355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" event={"ID":"833ca489-0771-4d2a-bc26-acf4cf476b1b","Type":"ContainerStarted","Data":"7b37ce661723a9944a5e48d0b8c64df2c09dd33c6e9081098719b3115b418db5"}
Sep 17 23:39:55.158021 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:55.157975    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" podStartSLOduration=1.157959826 podStartE2EDuration="1.157959826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:55.15638852 +0000 UTC m=+1755.608220009" watchObservedRunningTime="2026-09-17 23:39:55.157959826 +0000 UTC m=+1755.609791313"
Sep 17 23:39:55.390800 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:39:55.390737    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:40:04.390426 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:40:04.390387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:40:04.403476 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:40:04.403450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:40:05.201093 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:40:05.201056    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:40:40.122478 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:40:40.122447    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:40:40.127483 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:40:40.127462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:40:40.130554 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:40:40.130534    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:40:40.135220 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:40:40.135202    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:41:21.095498 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.095465    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"]
Sep 17 23:41:21.098850 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.098834    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.102665 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.102634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-djhs8\""
Sep 17 23:41:21.102802 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.102686    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 23:41:21.110742 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.110713    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"]
Sep 17 23:41:21.197899 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.197869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6dp5\" (UniqueName: \"kubernetes.io/projected/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-kube-api-access-x6dp5\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.198044 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.197923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.299189 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.299163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6dp5\" (UniqueName: \"kubernetes.io/projected/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-kube-api-access-x6dp5\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.299302 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.299206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.301548 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.301526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.307243 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.307219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6dp5\" (UniqueName: \"kubernetes.io/projected/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-kube-api-access-x6dp5\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.411428 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.411374    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:21.683124 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:21.683053    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"]
Sep 17 23:41:21.686272 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:41:21.686242    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded93f7d8_4405_42d6_857c_0f6d9c7b9d50.slice/crio-19b04fd62f8aaaa89c9c2dcef00eade6b1ff2801c3fccf57569cb5793a935129 WatchSource:0}: Error finding container 19b04fd62f8aaaa89c9c2dcef00eade6b1ff2801c3fccf57569cb5793a935129: Status 404 returned error can't find the container with id 19b04fd62f8aaaa89c9c2dcef00eade6b1ff2801c3fccf57569cb5793a935129
Sep 17 23:41:22.553431 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:22.553394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" event={"ID":"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50","Type":"ContainerStarted","Data":"d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549"}
Sep 17 23:41:22.553937 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:22.553450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" event={"ID":"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50","Type":"ContainerStarted","Data":"19b04fd62f8aaaa89c9c2dcef00eade6b1ff2801c3fccf57569cb5793a935129"}
Sep 17 23:41:22.553937 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:22.553523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:41:22.569315 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:22.569270    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" podStartSLOduration=1.5692563050000001 podStartE2EDuration="1.569256305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:41:22.568263625 +0000 UTC m=+1843.020095129" watchObservedRunningTime="2026-09-17 23:41:22.569256305 +0000 UTC m=+1843.021087795"
Sep 17 23:41:53.559835 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:41:53.559804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:42:04.311529 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:04.311442    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"]
Sep 17 23:42:13.559663 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:13.559617    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:42:19.333194 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:19.333153    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main" containerID="cri-o://d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549" gracePeriod=45
Sep 17 23:42:23.559829 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:23.559784    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:42:33.560458 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:33.560418    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:42:33.561047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:33.560542    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:42:43.559844 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:43.559801    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:42:49.578717 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.578690    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:42:49.652788 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.652724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6dp5\" (UniqueName: \"kubernetes.io/projected/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-kube-api-access-x6dp5\") pod \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") "
Sep 17 23:42:49.652788 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.652756    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-tls-certs\") pod \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\" (UID: \"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50\") "
Sep 17 23:42:49.654932 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.654901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-kube-api-access-x6dp5" (OuterVolumeSpecName: "kube-api-access-x6dp5") pod "ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" (UID: "ed93f7d8-4405-42d6-857c-0f6d9c7b9d50"). InnerVolumeSpecName "kube-api-access-x6dp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:49.655033 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.654977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" (UID: "ed93f7d8-4405-42d6-857c-0f6d9c7b9d50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:49.753623 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.753599    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6dp5\" (UniqueName: \"kubernetes.io/projected/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-kube-api-access-x6dp5\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:49.753623 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.753619    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:49.977261 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.977195    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerID="d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549" exitCode=0
Sep 17 23:42:49.977261 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.977243    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"
Sep 17 23:42:49.977443 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.977284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" event={"ID":"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50","Type":"ContainerDied","Data":"d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549"}
Sep 17 23:42:49.977443 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.977345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr" event={"ID":"ed93f7d8-4405-42d6-857c-0f6d9c7b9d50","Type":"ContainerDied","Data":"19b04fd62f8aaaa89c9c2dcef00eade6b1ff2801c3fccf57569cb5793a935129"}
Sep 17 23:42:49.977443 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.977375    2566 scope.go:117] "RemoveContainer" containerID="d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549"
Sep 17 23:42:49.988670 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.988648    2566 scope.go:117] "RemoveContainer" containerID="d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549"
Sep 17 23:42:49.989146 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:42:49.989123    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549\": container with ID starting with d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549 not found: ID does not exist" containerID="d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549"
Sep 17 23:42:49.989252 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.989152    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549"} err="failed to get container status \"d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549\": rpc error: code = NotFound desc = could not find container \"d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549\": container with ID starting with d24879ff75f5e97016c8c2333f12ca5bfbcd937522f5a5f83b62ba0d6e75b549 not found: ID does not exist"
Sep 17 23:42:49.998076 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:49.998054    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"]
Sep 17 23:42:50.001834 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:50.001816    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9kqwr"]
Sep 17 23:42:50.013648 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:50.013626    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" path="/var/lib/kubelet/pods/ed93f7d8-4405-42d6-857c-0f6d9c7b9d50/volumes"
Sep 17 23:42:53.628234 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.628200    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"]
Sep 17 23:42:53.638930 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:42:53.638879    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:42:53.638930 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:42:53.638930 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:42:53.638930 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c" cmd=["/bin/sleep","15"]
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.638954    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" message="" err=<
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:42:53.638970    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" podUID="833ca489-0771-4d2a-bc26-acf4cf476b1b" containerName="main" containerID="cri-o://074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c"
Sep 17 23:42:53.639232 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.639015    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" podUID="833ca489-0771-4d2a-bc26-acf4cf476b1b" containerName="main" containerID="cri-o://074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c" gracePeriod=60
Sep 17 23:42:53.986648 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.986618    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:42:53.997142 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.997110    2566 generic.go:356] "Generic (PLEG): container finished" podID="833ca489-0771-4d2a-bc26-acf4cf476b1b" containerID="074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c" exitCode=0
Sep 17 23:42:53.997255 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.997153    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"
Sep 17 23:42:53.997255 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.997168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" event={"ID":"833ca489-0771-4d2a-bc26-acf4cf476b1b","Type":"ContainerDied","Data":"074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c"}
Sep 17 23:42:53.997255 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.997211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99" event={"ID":"833ca489-0771-4d2a-bc26-acf4cf476b1b","Type":"ContainerDied","Data":"7b37ce661723a9944a5e48d0b8c64df2c09dd33c6e9081098719b3115b418db5"}
Sep 17 23:42:53.997255 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:53.997237    2566 scope.go:117] "RemoveContainer" containerID="074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c"
Sep 17 23:42:54.006357 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.006335    2566 scope.go:117] "RemoveContainer" containerID="074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c"
Sep 17 23:42:54.006775 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:42:54.006739    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c\": container with ID starting with 074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c not found: ID does not exist" containerID="074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c"
Sep 17 23:42:54.006934 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.006902    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c"} err="failed to get container status \"074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c\": rpc error: code = NotFound desc = could not find container \"074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c\": container with ID starting with 074b1583c16eba9107025ba6cc7b74f440494ae45c7db8502a403fabc498432c not found: ID does not exist"
Sep 17 23:42:54.086631 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086601    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-tmp-dir\") pod \"833ca489-0771-4d2a-bc26-acf4cf476b1b\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") "
Sep 17 23:42:54.086772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833ca489-0771-4d2a-bc26-acf4cf476b1b-tls-certs\") pod \"833ca489-0771-4d2a-bc26-acf4cf476b1b\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") "
Sep 17 23:42:54.086772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-home\") pod \"833ca489-0771-4d2a-bc26-acf4cf476b1b\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") "
Sep 17 23:42:54.086772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wngqm\" (UniqueName: \"kubernetes.io/projected/833ca489-0771-4d2a-bc26-acf4cf476b1b-kube-api-access-wngqm\") pod \"833ca489-0771-4d2a-bc26-acf4cf476b1b\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") "
Sep 17 23:42:54.086772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-model-cache\") pod \"833ca489-0771-4d2a-bc26-acf4cf476b1b\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") "
Sep 17 23:42:54.087001 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086799    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-dshm\") pod \"833ca489-0771-4d2a-bc26-acf4cf476b1b\" (UID: \"833ca489-0771-4d2a-bc26-acf4cf476b1b\") "
Sep 17 23:42:54.087001 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086876    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "833ca489-0771-4d2a-bc26-acf4cf476b1b" (UID: "833ca489-0771-4d2a-bc26-acf4cf476b1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:54.087001 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.086960    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-home" (OuterVolumeSpecName: "home") pod "833ca489-0771-4d2a-bc26-acf4cf476b1b" (UID: "833ca489-0771-4d2a-bc26-acf4cf476b1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:54.087150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.087055    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "833ca489-0771-4d2a-bc26-acf4cf476b1b" (UID: "833ca489-0771-4d2a-bc26-acf4cf476b1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:54.087150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.087116    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:54.087150 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.087135    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:54.088823 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.088797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-dshm" (OuterVolumeSpecName: "dshm") pod "833ca489-0771-4d2a-bc26-acf4cf476b1b" (UID: "833ca489-0771-4d2a-bc26-acf4cf476b1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:54.088926 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.088893    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833ca489-0771-4d2a-bc26-acf4cf476b1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "833ca489-0771-4d2a-bc26-acf4cf476b1b" (UID: "833ca489-0771-4d2a-bc26-acf4cf476b1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:54.088926 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.088903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833ca489-0771-4d2a-bc26-acf4cf476b1b-kube-api-access-wngqm" (OuterVolumeSpecName: "kube-api-access-wngqm") pod "833ca489-0771-4d2a-bc26-acf4cf476b1b" (UID: "833ca489-0771-4d2a-bc26-acf4cf476b1b"). InnerVolumeSpecName "kube-api-access-wngqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:54.187505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.187445    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:54.187505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.187464    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833ca489-0771-4d2a-bc26-acf4cf476b1b-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:54.187505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.187473    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833ca489-0771-4d2a-bc26-acf4cf476b1b-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:54.187505 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.187481    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngqm\" (UniqueName: \"kubernetes.io/projected/833ca489-0771-4d2a-bc26-acf4cf476b1b-kube-api-access-wngqm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:42:54.318204 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.318176    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"]
Sep 17 23:42:54.322415 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:54.322393    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-rjn99"]
Sep 17 23:42:56.013331 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:42:56.013299    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833ca489-0771-4d2a-bc26-acf4cf476b1b" path="/var/lib/kubelet/pods/833ca489-0771-4d2a-bc26-acf4cf476b1b/volumes"
Sep 17 23:44:01.099272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.099224    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"]
Sep 17 23:44:01.099902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.099807    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main"
Sep 17 23:44:01.099902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.099831    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main"
Sep 17 23:44:01.099902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.099867    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833ca489-0771-4d2a-bc26-acf4cf476b1b" containerName="main"
Sep 17 23:44:01.099902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.099877    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="833ca489-0771-4d2a-bc26-acf4cf476b1b" containerName="main"
Sep 17 23:44:01.100126 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.099960    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="833ca489-0771-4d2a-bc26-acf4cf476b1b" containerName="main"
Sep 17 23:44:01.100126 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.099976    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed93f7d8-4405-42d6-857c-0f6d9c7b9d50" containerName="main"
Sep 17 23:44:01.103098 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.103075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.107525 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.107498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 23:44:01.107676 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.107509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 23:44:01.107676 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.107509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-6dldf\""
Sep 17 23:44:01.107676 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.107616    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 23:44:01.112468 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.112440    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"]
Sep 17 23:44:01.184798 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.184768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.184925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.184803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.184925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.184859    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/f78656ed-5692-4e17-8ea8-c62a0d45b337-kube-api-access-m2hnc\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.184925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.184886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.184925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.184909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.185106 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.184963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78656ed-5692-4e17-8ea8-c62a0d45b337-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.285824 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.285800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/f78656ed-5692-4e17-8ea8-c62a0d45b337-kube-api-access-m2hnc\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.285940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.285829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.285940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.285850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.285940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.285881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78656ed-5692-4e17-8ea8-c62a0d45b337-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.285940 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.285932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.286167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.286048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.286272 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.286254    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.286327 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.286289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.286422 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.286334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.288196 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.288175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.288444 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.288425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78656ed-5692-4e17-8ea8-c62a0d45b337-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.299708 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.299688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/f78656ed-5692-4e17-8ea8-c62a0d45b337-kube-api-access-m2hnc\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.348629 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.348604    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"]
Sep 17 23:44:01.352560 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.352512    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.355100 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.355079    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-jw7h5\""
Sep 17 23:44:01.359140 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.359119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"]
Sep 17 23:44:01.386651 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.386613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6s8\" (UniqueName: \"kubernetes.io/projected/b9b3d470-8738-4611-9a7e-a9b757cc00e8-kube-api-access-lq6s8\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.386771 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.386659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b3d470-8738-4611-9a7e-a9b757cc00e8-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.415644 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.415622    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:01.487725 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.487690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6s8\" (UniqueName: \"kubernetes.io/projected/b9b3d470-8738-4611-9a7e-a9b757cc00e8-kube-api-access-lq6s8\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.487863 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.487762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b3d470-8738-4611-9a7e-a9b757cc00e8-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.490167 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.490142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b3d470-8738-4611-9a7e-a9b757cc00e8-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.496534 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.496511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6s8\" (UniqueName: \"kubernetes.io/projected/b9b3d470-8738-4611-9a7e-a9b757cc00e8-kube-api-access-lq6s8\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.665296 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.665230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:01.688828 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.688803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"]
Sep 17 23:44:01.690391 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:44:01.690363    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78656ed_5692_4e17_8ea8_c62a0d45b337.slice/crio-199e28307e34e188fbab138c3abda5299a7714afd3f2f3dcec22f7502ffcc4d2 WatchSource:0}: Error finding container 199e28307e34e188fbab138c3abda5299a7714afd3f2f3dcec22f7502ffcc4d2: Status 404 returned error can't find the container with id 199e28307e34e188fbab138c3abda5299a7714afd3f2f3dcec22f7502ffcc4d2
Sep 17 23:44:01.942041 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:01.942011    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"]
Sep 17 23:44:01.943384 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:44:01.943358    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b3d470_8738_4611_9a7e_a9b757cc00e8.slice/crio-740b86cd682f1b860dcceb6d338dea2b7c95877dca7bfdadc981f6891a46e008 WatchSource:0}: Error finding container 740b86cd682f1b860dcceb6d338dea2b7c95877dca7bfdadc981f6891a46e008: Status 404 returned error can't find the container with id 740b86cd682f1b860dcceb6d338dea2b7c95877dca7bfdadc981f6891a46e008
Sep 17 23:44:02.327681 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.327643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" event={"ID":"b9b3d470-8738-4611-9a7e-a9b757cc00e8","Type":"ContainerStarted","Data":"55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e"}
Sep 17 23:44:02.328129 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.327739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" event={"ID":"b9b3d470-8738-4611-9a7e-a9b757cc00e8","Type":"ContainerStarted","Data":"740b86cd682f1b860dcceb6d338dea2b7c95877dca7bfdadc981f6891a46e008"}
Sep 17 23:44:02.328129 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.327776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:44:02.329283 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.329257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" event={"ID":"f78656ed-5692-4e17-8ea8-c62a0d45b337","Type":"ContainerStarted","Data":"5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2"}
Sep 17 23:44:02.329419 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.329307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" event={"ID":"f78656ed-5692-4e17-8ea8-c62a0d45b337","Type":"ContainerStarted","Data":"199e28307e34e188fbab138c3abda5299a7714afd3f2f3dcec22f7502ffcc4d2"}
Sep 17 23:44:02.346097 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.346056    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" podStartSLOduration=1.346044647 podStartE2EDuration="1.346044647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:02.344040653 +0000 UTC m=+2002.795872182" watchObservedRunningTime="2026-09-17 23:44:02.346044647 +0000 UTC m=+2002.797876126"
Sep 17 23:44:02.366412 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.366359    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" podStartSLOduration=1.3663420419999999 podStartE2EDuration="1.366342042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:02.365766034 +0000 UTC m=+2002.817597518" watchObservedRunningTime="2026-09-17 23:44:02.366342042 +0000 UTC m=+2002.818173532"
Sep 17 23:44:02.416368 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:02.416345    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:11.416749 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:11.416706    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:11.429284 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:11.429257    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:12.390449 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:12.390418    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:44:33.335408 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:44:33.335366    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:45:40.154636 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:45:40.154608    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:45:40.159513 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:45:40.159476    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:45:40.170076 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:45:40.170044    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:45:40.173827 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:45:40.173809    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:46:09.845831 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:09.845789    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"]
Sep 17 23:46:09.848398 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:09.848368    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"]
Sep 17 23:46:09.855783 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:46:09.855742    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:46:09.855783 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:46:09.855783 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:46:09.855783 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2" cmd=["/bin/sleep","15"]
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:09.855806    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" message="" err=<
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:46:09.855819    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" podUID="f78656ed-5692-4e17-8ea8-c62a0d45b337" containerName="main" containerID="cri-o://5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2"
Sep 17 23:46:09.856079 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:09.855879    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" podUID="f78656ed-5692-4e17-8ea8-c62a0d45b337" containerName="main" containerID="cri-o://5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2" gracePeriod=60
Sep 17 23:46:10.224646 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.224616    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:46:10.266403 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266371    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/f78656ed-5692-4e17-8ea8-c62a0d45b337-kube-api-access-m2hnc\") pod \"f78656ed-5692-4e17-8ea8-c62a0d45b337\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") "
Sep 17 23:46:10.266603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266429    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-tmp-dir\") pod \"f78656ed-5692-4e17-8ea8-c62a0d45b337\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") "
Sep 17 23:46:10.266603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78656ed-5692-4e17-8ea8-c62a0d45b337-tls-certs\") pod \"f78656ed-5692-4e17-8ea8-c62a0d45b337\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") "
Sep 17 23:46:10.266603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-dshm\") pod \"f78656ed-5692-4e17-8ea8-c62a0d45b337\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") "
Sep 17 23:46:10.266603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-model-cache\") pod \"f78656ed-5692-4e17-8ea8-c62a0d45b337\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") "
Sep 17 23:46:10.266603 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266543    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-home\") pod \"f78656ed-5692-4e17-8ea8-c62a0d45b337\" (UID: \"f78656ed-5692-4e17-8ea8-c62a0d45b337\") "
Sep 17 23:46:10.266926 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266730    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f78656ed-5692-4e17-8ea8-c62a0d45b337" (UID: "f78656ed-5692-4e17-8ea8-c62a0d45b337"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:46:10.266926 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266846    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:10.266926 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-model-cache" (OuterVolumeSpecName: "model-cache") pod "f78656ed-5692-4e17-8ea8-c62a0d45b337" (UID: "f78656ed-5692-4e17-8ea8-c62a0d45b337"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:46:10.267094 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.266967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-home" (OuterVolumeSpecName: "home") pod "f78656ed-5692-4e17-8ea8-c62a0d45b337" (UID: "f78656ed-5692-4e17-8ea8-c62a0d45b337"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:46:10.268772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.268742    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78656ed-5692-4e17-8ea8-c62a0d45b337-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f78656ed-5692-4e17-8ea8-c62a0d45b337" (UID: "f78656ed-5692-4e17-8ea8-c62a0d45b337"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:46:10.269008 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.268987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-dshm" (OuterVolumeSpecName: "dshm") pod "f78656ed-5692-4e17-8ea8-c62a0d45b337" (UID: "f78656ed-5692-4e17-8ea8-c62a0d45b337"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:46:10.269094 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.269072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78656ed-5692-4e17-8ea8-c62a0d45b337-kube-api-access-m2hnc" (OuterVolumeSpecName: "kube-api-access-m2hnc") pod "f78656ed-5692-4e17-8ea8-c62a0d45b337" (UID: "f78656ed-5692-4e17-8ea8-c62a0d45b337"). InnerVolumeSpecName "kube-api-access-m2hnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:46:10.367523 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.367488    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:10.367523 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.367517    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:10.367523 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.367529    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78656ed-5692-4e17-8ea8-c62a0d45b337-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:10.367747 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.367540    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/f78656ed-5692-4e17-8ea8-c62a0d45b337-kube-api-access-m2hnc\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:10.367747 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.367551    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78656ed-5692-4e17-8ea8-c62a0d45b337-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:10.935984 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.935945    2566 generic.go:356] "Generic (PLEG): container finished" podID="f78656ed-5692-4e17-8ea8-c62a0d45b337" containerID="5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2" exitCode=0
Sep 17 23:46:10.936389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.935989    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"
Sep 17 23:46:10.936389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.936035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" event={"ID":"f78656ed-5692-4e17-8ea8-c62a0d45b337","Type":"ContainerDied","Data":"5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2"}
Sep 17 23:46:10.936389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.936091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp" event={"ID":"f78656ed-5692-4e17-8ea8-c62a0d45b337","Type":"ContainerDied","Data":"199e28307e34e188fbab138c3abda5299a7714afd3f2f3dcec22f7502ffcc4d2"}
Sep 17 23:46:10.936389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.936115    2566 scope.go:117] "RemoveContainer" containerID="5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2"
Sep 17 23:46:10.946773 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.946753    2566 scope.go:117] "RemoveContainer" containerID="5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2"
Sep 17 23:46:10.947060 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:46:10.947040    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2\": container with ID starting with 5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2 not found: ID does not exist" containerID="5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2"
Sep 17 23:46:10.947137 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.947068    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2"} err="failed to get container status \"5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2\": rpc error: code = NotFound desc = could not find container \"5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2\": container with ID starting with 5c71b17a8df6a20029363f2d805bb94ce8e901d8a1616b9b4186867f1e3d82b2 not found: ID does not exist"
Sep 17 23:46:10.962589 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.962553    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"]
Sep 17 23:46:10.964105 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:10.964086    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969jjwdp"]
Sep 17 23:46:12.013896 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:12.013861    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f78656ed-5692-4e17-8ea8-c62a0d45b337" path="/var/lib/kubelet/pods/f78656ed-5692-4e17-8ea8-c62a0d45b337/volumes"
Sep 17 23:46:13.335705 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:13.335666    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:46:23.335341 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:23.335291    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:46:24.870121 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:24.870079    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main" containerID="cri-o://55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e" gracePeriod=45
Sep 17 23:46:33.335556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:33.335458    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:46:33.336018 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:33.335636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:46:43.336497 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:43.336437    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:46:53.335917 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:53.335869    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:46:55.110441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.110416    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:46:55.157559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.157499    2566 generic.go:356] "Generic (PLEG): container finished" podID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerID="55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e" exitCode=0
Sep 17 23:46:55.157559 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.157542    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"
Sep 17 23:46:55.157730 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.157559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" event={"ID":"b9b3d470-8738-4611-9a7e-a9b757cc00e8","Type":"ContainerDied","Data":"55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e"}
Sep 17 23:46:55.157730 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.157610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh" event={"ID":"b9b3d470-8738-4611-9a7e-a9b757cc00e8","Type":"ContainerDied","Data":"740b86cd682f1b860dcceb6d338dea2b7c95877dca7bfdadc981f6891a46e008"}
Sep 17 23:46:55.157730 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.157631    2566 scope.go:117] "RemoveContainer" containerID="55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e"
Sep 17 23:46:55.165760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.165739    2566 scope.go:117] "RemoveContainer" containerID="55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e"
Sep 17 23:46:55.166069 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:46:55.166040    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e\": container with ID starting with 55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e not found: ID does not exist" containerID="55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e"
Sep 17 23:46:55.166148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.166071    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e"} err="failed to get container status \"55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e\": rpc error: code = NotFound desc = could not find container \"55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e\": container with ID starting with 55978a06bf2af832aacb0cb30ae2e2acb92e8e2b1aff6996e5f71264ef674a0e not found: ID does not exist"
Sep 17 23:46:55.192801 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.192779    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq6s8\" (UniqueName: \"kubernetes.io/projected/b9b3d470-8738-4611-9a7e-a9b757cc00e8-kube-api-access-lq6s8\") pod \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") "
Sep 17 23:46:55.192872 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.192834    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b3d470-8738-4611-9a7e-a9b757cc00e8-tls-certs\") pod \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\" (UID: \"b9b3d470-8738-4611-9a7e-a9b757cc00e8\") "
Sep 17 23:46:55.194755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.194733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b3d470-8738-4611-9a7e-a9b757cc00e8-kube-api-access-lq6s8" (OuterVolumeSpecName: "kube-api-access-lq6s8") pod "b9b3d470-8738-4611-9a7e-a9b757cc00e8" (UID: "b9b3d470-8738-4611-9a7e-a9b757cc00e8"). InnerVolumeSpecName "kube-api-access-lq6s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:46:55.194829 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.194775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b3d470-8738-4611-9a7e-a9b757cc00e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9b3d470-8738-4611-9a7e-a9b757cc00e8" (UID: "b9b3d470-8738-4611-9a7e-a9b757cc00e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:46:55.293687 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.293663    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b3d470-8738-4611-9a7e-a9b757cc00e8-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:55.293687 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.293686    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6s8\" (UniqueName: \"kubernetes.io/projected/b9b3d470-8738-4611-9a7e-a9b757cc00e8-kube-api-access-lq6s8\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:46:55.479682 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.479654    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"]
Sep 17 23:46:55.485032 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:55.485008    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5rkhh"]
Sep 17 23:46:56.014617 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:46:56.014558    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" path="/var/lib/kubelet/pods/b9b3d470-8738-4611-9a7e-a9b757cc00e8/volumes"
Sep 17 23:47:14.912969 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.912933    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"]
Sep 17 23:47:14.913404 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.913294    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main"
Sep 17 23:47:14.913404 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.913307    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main"
Sep 17 23:47:14.913404 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.913333    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78656ed-5692-4e17-8ea8-c62a0d45b337" containerName="main"
Sep 17 23:47:14.913404 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.913339    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78656ed-5692-4e17-8ea8-c62a0d45b337" containerName="main"
Sep 17 23:47:14.913404 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.913407    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78656ed-5692-4e17-8ea8-c62a0d45b337" containerName="main"
Sep 17 23:47:14.913569 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.913416    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b3d470-8738-4611-9a7e-a9b757cc00e8" containerName="main"
Sep 17 23:47:14.917916 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.917897    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:14.920532 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.920506    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 23:47:14.920877 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.920856    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:47:14.920999 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.920960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-sdbwq\""
Sep 17 23:47:14.921117 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.920973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 23:47:14.921117 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.921045    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 23:47:14.921726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.921709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 23:47:14.926936 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:14.926911    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"]
Sep 17 23:47:15.048861 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.048831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.049004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.048883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6227f2-ff20-4485-94db-3b9cf5a6d275-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.049004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.048941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.049004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.048972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.049128 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.049026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.049128 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.049048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.049128 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.049105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.049225 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.049151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpv6g\" (UniqueName: \"kubernetes.io/projected/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kube-api-access-kpv6g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150208 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150359 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150440 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150510 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150510 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpv6g\" (UniqueName: \"kubernetes.io/projected/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kube-api-access-kpv6g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150657 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150657 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6227f2-ff20-4485-94db-3b9cf5a6d275-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150766 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150766 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.150766 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.150753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.151090 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.151061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.151400 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.151374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.151662 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.151635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6227f2-ff20-4485-94db-3b9cf5a6d275-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.153755 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.153728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.154738 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.154710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.156726 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.156696    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"]
Sep 17 23:47:15.160535 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.160513    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.163025 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.162952    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-4t8p6\""
Sep 17 23:47:15.163595 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.163548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpv6g\" (UniqueName: \"kubernetes.io/projected/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kube-api-access-kpv6g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.168957 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.168934    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"]
Sep 17 23:47:15.230447 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.230424    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:15.252154 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.252121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxw8\" (UniqueName: \"kubernetes.io/projected/4e986442-f285-424e-af9b-6902d602f93f-kube-api-access-2kxw8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.252154 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.252152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e986442-f285-424e-af9b-6902d602f93f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.353389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.353347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxw8\" (UniqueName: \"kubernetes.io/projected/4e986442-f285-424e-af9b-6902d602f93f-kube-api-access-2kxw8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.353389 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.353394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e986442-f285-424e-af9b-6902d602f93f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.355750 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.355726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e986442-f285-424e-af9b-6902d602f93f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.366201 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.363066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxw8\" (UniqueName: \"kubernetes.io/projected/4e986442-f285-424e-af9b-6902d602f93f-kube-api-access-2kxw8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.489765 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.489742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:15.509818 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.509791    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"]
Sep 17 23:47:15.512425 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:47:15.512400    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6227f2_ff20_4485_94db_3b9cf5a6d275.slice/crio-372db63daceee2993f550f5034459f2b63321d9933abb0fe6998a72b012d1d76 WatchSource:0}: Error finding container 372db63daceee2993f550f5034459f2b63321d9933abb0fe6998a72b012d1d76: Status 404 returned error can't find the container with id 372db63daceee2993f550f5034459f2b63321d9933abb0fe6998a72b012d1d76
Sep 17 23:47:15.760239 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:15.760162    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"]
Sep 17 23:47:15.763320 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:47:15.763292    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e986442_f285_424e_af9b_6902d602f93f.slice/crio-94d1050bf230a4d17bea96f706b99b32bc326da5aaad96e3b0a3a2df1e197ba8 WatchSource:0}: Error finding container 94d1050bf230a4d17bea96f706b99b32bc326da5aaad96e3b0a3a2df1e197ba8: Status 404 returned error can't find the container with id 94d1050bf230a4d17bea96f706b99b32bc326da5aaad96e3b0a3a2df1e197ba8
Sep 17 23:47:16.259661 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:16.259612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" event={"ID":"4e986442-f285-424e-af9b-6902d602f93f","Type":"ContainerStarted","Data":"de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e"}
Sep 17 23:47:16.260105 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:16.259683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" event={"ID":"4e986442-f285-424e-af9b-6902d602f93f","Type":"ContainerStarted","Data":"94d1050bf230a4d17bea96f706b99b32bc326da5aaad96e3b0a3a2df1e197ba8"}
Sep 17 23:47:16.260105 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:16.259727    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:16.261924 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:16.261891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" event={"ID":"8d6227f2-ff20-4485-94db-3b9cf5a6d275","Type":"ContainerStarted","Data":"6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd"}
Sep 17 23:47:16.262058 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:16.261951    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" event={"ID":"8d6227f2-ff20-4485-94db-3b9cf5a6d275","Type":"ContainerStarted","Data":"372db63daceee2993f550f5034459f2b63321d9933abb0fe6998a72b012d1d76"}
Sep 17 23:47:16.279107 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:16.279053    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" podStartSLOduration=1.27903519 podStartE2EDuration="1.27903519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:47:16.278026672 +0000 UTC m=+2196.729858161" watchObservedRunningTime="2026-09-17 23:47:16.27903519 +0000 UTC m=+2196.730866676"
Sep 17 23:47:19.279803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:19.279768    2566 generic.go:356] "Generic (PLEG): container finished" podID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerID="6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd" exitCode=0
Sep 17 23:47:19.280207 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:19.279858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" event={"ID":"8d6227f2-ff20-4485-94db-3b9cf5a6d275","Type":"ContainerDied","Data":"6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd"}
Sep 17 23:47:19.281237 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:19.281220    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:47:47.269432 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:47.269401    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:47:48.460120 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:48.460083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" event={"ID":"8d6227f2-ff20-4485-94db-3b9cf5a6d275","Type":"ContainerStarted","Data":"e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f"}
Sep 17 23:47:48.482852 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:48.482805    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podStartSLOduration=3.3190047910000002 podStartE2EDuration="34.4827893s" totalImagesPullingTime="28.288380749s" totalInitContainerRuntime="2.87540376s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:47:48.480196043 +0000 UTC m=+2228.932027530" watchObservedRunningTime="2026-09-17 23:47:48.4827893 +0000 UTC m=+2228.934620787"
Sep 17 23:47:49.231370 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:49.231334    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:55.231482 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:55.231445    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:47:55.232717 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:47:55.232688    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:48:05.231177 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:48:05.231089    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:48:15.230886 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:48:15.230844    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:48:25.231297 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:48:25.231251    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:48:35.230828 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:48:35.230774    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:48:45.231733 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:48:45.231693    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:48:55.231149 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:48:55.231101    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:49:05.241843 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:49:05.241807    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:49:05.249889 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:49:05.249857    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:49:47.964443 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:49:47.964222    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"]
Sep 17 23:49:47.966496 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:49:47.966459    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"]
Sep 17 23:49:57.269084 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:49:57.269041    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:50:02.985584 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:02.985527    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main" containerID="cri-o://de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e" gracePeriod=45
Sep 17 23:50:02.987169 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:02.987122    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" containerID="cri-o://e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f" gracePeriod=45
Sep 17 23:50:03.242838 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.242747    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 23:50:03.638218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.638195    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:50:03.663105 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-dshm\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663251 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663131    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-model-cache\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663251 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663178    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tmp-dir\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663251 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663204    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpv6g\" (UniqueName: \"kubernetes.io/projected/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kube-api-access-kpv6g\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663251 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663242    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kserve-provision-location\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-home\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6227f2-ff20-4485-94db-3b9cf5a6d275-cabundle-cert\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663391    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tls-certs\") pod \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\" (UID: \"8d6227f2-ff20-4485-94db-3b9cf5a6d275\") "
Sep 17 23:50:03.663481 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:50:03.663719 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663674    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:03.663925 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-home" (OuterVolumeSpecName: "home") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:50:03.664026 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.663995    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6227f2-ff20-4485-94db-3b9cf5a6d275-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:50:03.665902 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.665881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-dshm" (OuterVolumeSpecName: "dshm") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:50:03.666141 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.666102    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:50:03.668131 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.668093    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kube-api-access-kpv6g" (OuterVolumeSpecName: "kube-api-access-kpv6g") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "kube-api-access-kpv6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:50:03.675597 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.675554    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:50:03.693220 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.693197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d6227f2-ff20-4485-94db-3b9cf5a6d275" (UID: "8d6227f2-ff20-4485-94db-3b9cf5a6d275"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:50:03.764817 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.764769    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d6227f2-ff20-4485-94db-3b9cf5a6d275-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:03.764817 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.764788    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:03.764817 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.764796    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:03.764817 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.764804    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:03.764817 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.764813    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpv6g\" (UniqueName: \"kubernetes.io/projected/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kube-api-access-kpv6g\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:03.765015 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.764822    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:03.765015 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:03.764830    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6227f2-ff20-4485-94db-3b9cf5a6d275-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:04.128698 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.128668    2566 generic.go:356] "Generic (PLEG): container finished" podID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerID="e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f" exitCode=0
Sep 17 23:50:04.129063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.128727    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"
Sep 17 23:50:04.129063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.128756    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" event={"ID":"8d6227f2-ff20-4485-94db-3b9cf5a6d275","Type":"ContainerDied","Data":"e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f"}
Sep 17 23:50:04.129063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.128811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b" event={"ID":"8d6227f2-ff20-4485-94db-3b9cf5a6d275","Type":"ContainerDied","Data":"372db63daceee2993f550f5034459f2b63321d9933abb0fe6998a72b012d1d76"}
Sep 17 23:50:04.129063 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.128835    2566 scope.go:117] "RemoveContainer" containerID="e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f"
Sep 17 23:50:04.137468 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.137438    2566 scope.go:117] "RemoveContainer" containerID="6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd"
Sep 17 23:50:04.145953 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.145910    2566 scope.go:117] "RemoveContainer" containerID="e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f"
Sep 17 23:50:04.146286 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:50:04.146263    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f\": container with ID starting with e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f not found: ID does not exist" containerID="e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f"
Sep 17 23:50:04.146382 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.146293    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f"} err="failed to get container status \"e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f\": rpc error: code = NotFound desc = could not find container \"e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f\": container with ID starting with e1c322c6269b4b5ba94a6099e69ff9ec2f53a8ae8b043dc34121dadf2bfcd11f not found: ID does not exist"
Sep 17 23:50:04.146382 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.146308    2566 scope.go:117] "RemoveContainer" containerID="6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd"
Sep 17 23:50:04.147071 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:50:04.146662    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd\": container with ID starting with 6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd not found: ID does not exist" containerID="6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd"
Sep 17 23:50:04.147071 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.146772    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd"} err="failed to get container status \"6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd\": rpc error: code = NotFound desc = could not find container \"6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd\": container with ID starting with 6c655b8e9de64c38411202c16e0c85b30f60beefa9daf1a721b020ad93dae3dd not found: ID does not exist"
Sep 17 23:50:04.148247 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.148227    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"]
Sep 17 23:50:04.153096 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:04.153075    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5857b85c77-8bb8b"]
Sep 17 23:50:06.014758 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:06.014715    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" path="/var/lib/kubelet/pods/8d6227f2-ff20-4485-94db-3b9cf5a6d275/volumes"
Sep 17 23:50:07.269855 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:07.269816    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:50:17.269757 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:17.269719    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:50:17.270280 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:17.269830    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:50:27.269650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:27.269615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:50:33.233751 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.233725    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:50:33.273496 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.273433    2566 generic.go:356] "Generic (PLEG): container finished" podID="4e986442-f285-424e-af9b-6902d602f93f" containerID="de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e" exitCode=0
Sep 17 23:50:33.273633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.273487    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"
Sep 17 23:50:33.273633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.273503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" event={"ID":"4e986442-f285-424e-af9b-6902d602f93f","Type":"ContainerDied","Data":"de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e"}
Sep 17 23:50:33.273633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.273548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4" event={"ID":"4e986442-f285-424e-af9b-6902d602f93f","Type":"ContainerDied","Data":"94d1050bf230a4d17bea96f706b99b32bc326da5aaad96e3b0a3a2df1e197ba8"}
Sep 17 23:50:33.273633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.273570    2566 scope.go:117] "RemoveContainer" containerID="de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e"
Sep 17 23:50:33.281786 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.281769    2566 scope.go:117] "RemoveContainer" containerID="de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e"
Sep 17 23:50:33.282062 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:50:33.282044    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e\": container with ID starting with de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e not found: ID does not exist" containerID="de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e"
Sep 17 23:50:33.282130 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.282080    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e"} err="failed to get container status \"de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e\": rpc error: code = NotFound desc = could not find container \"de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e\": container with ID starting with de18bc806ab509c4636b31bdb078593715074c32b7da35a7f040c149d0679e7e not found: ID does not exist"
Sep 17 23:50:33.287364 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.287343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kxw8\" (UniqueName: \"kubernetes.io/projected/4e986442-f285-424e-af9b-6902d602f93f-kube-api-access-2kxw8\") pod \"4e986442-f285-424e-af9b-6902d602f93f\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") "
Sep 17 23:50:33.287438 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.287405    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e986442-f285-424e-af9b-6902d602f93f-tls-certs\") pod \"4e986442-f285-424e-af9b-6902d602f93f\" (UID: \"4e986442-f285-424e-af9b-6902d602f93f\") "
Sep 17 23:50:33.289236 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.289214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e986442-f285-424e-af9b-6902d602f93f-kube-api-access-2kxw8" (OuterVolumeSpecName: "kube-api-access-2kxw8") pod "4e986442-f285-424e-af9b-6902d602f93f" (UID: "4e986442-f285-424e-af9b-6902d602f93f"). InnerVolumeSpecName "kube-api-access-2kxw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:50:33.289338 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.289316    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e986442-f285-424e-af9b-6902d602f93f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e986442-f285-424e-af9b-6902d602f93f" (UID: "4e986442-f285-424e-af9b-6902d602f93f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:50:33.388029 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.388006    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e986442-f285-424e-af9b-6902d602f93f-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:33.388029 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.388026    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxw8\" (UniqueName: \"kubernetes.io/projected/4e986442-f285-424e-af9b-6902d602f93f-kube-api-access-2kxw8\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:50:33.597212 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.597128    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"]
Sep 17 23:50:33.603994 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:33.603969    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2srb4"]
Sep 17 23:50:34.014202 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:34.014170    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e986442-f285-424e-af9b-6902d602f93f" path="/var/lib/kubelet/pods/4e986442-f285-424e-af9b-6902d602f93f/volumes"
Sep 17 23:50:40.191327 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:40.191298    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:50:40.198334 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:40.198308    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:50:40.199064 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:40.199043    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:50:40.205981 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:40.205962    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:50:43.215959 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.215920    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"]
Sep 17 23:50:43.216348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216269    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="storage-initializer"
Sep 17 23:50:43.216348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216282    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="storage-initializer"
Sep 17 23:50:43.216348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216292    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main"
Sep 17 23:50:43.216348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216298    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main"
Sep 17 23:50:43.216348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216306    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main"
Sep 17 23:50:43.216348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216313    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main"
Sep 17 23:50:43.216562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216368    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e986442-f285-424e-af9b-6902d602f93f" containerName="main"
Sep 17 23:50:43.216562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.216376    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6227f2-ff20-4485-94db-3b9cf5a6d275" containerName="main"
Sep 17 23:50:43.220984 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.220962    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"]
Sep 17 23:50:43.221108 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.221050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.223810 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.223786    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:50:43.223935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.223786    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 23:50:43.223935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.223792    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 23:50:43.224053 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.223871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-g5k8r\""
Sep 17 23:50:43.224053 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.223985    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 23:50:43.224224 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.224207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 23:50:43.361082 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.361235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ce118e-5671-4647-adcd-72c9700ce69a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.361235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.361235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.361235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55c7\" (UniqueName: \"kubernetes.io/projected/71ce118e-5671-4647-adcd-72c9700ce69a-kube-api-access-z55c7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.361380 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.361380 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.361380 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.361298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ce118e-5671-4647-adcd-72c9700ce69a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.434516 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.434464    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"]
Sep 17 23:50:43.437851 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.437835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.440507 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.440484    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-stgrb\""
Sep 17 23:50:43.448428 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.448395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"]
Sep 17 23:50:43.461674 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.461803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.461803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ce118e-5671-4647-adcd-72c9700ce69a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.461803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.461803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ce118e-5671-4647-adcd-72c9700ce69a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462042 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462042 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462042 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.461860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55c7\" (UniqueName: \"kubernetes.io/projected/71ce118e-5671-4647-adcd-72c9700ce69a-kube-api-access-z55c7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462203 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.462098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462647 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.462623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462806 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.462778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462937 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.462738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ce118e-5671-4647-adcd-72c9700ce69a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.462937 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.462660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.464081 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.464058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.464675 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.464655    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ce118e-5671-4647-adcd-72c9700ce69a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.475007 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.474944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55c7\" (UniqueName: \"kubernetes.io/projected/71ce118e-5671-4647-adcd-72c9700ce69a-kube-api-access-z55c7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.531785 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.531756    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:43.563126 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.563095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrl9m\" (UniqueName: \"kubernetes.io/projected/53c6d6c9-1832-4e87-9cc9-c28268fd8154-kube-api-access-qrl9m\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.563289 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.563182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c6d6c9-1832-4e87-9cc9-c28268fd8154-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.664268 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.664233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrl9m\" (UniqueName: \"kubernetes.io/projected/53c6d6c9-1832-4e87-9cc9-c28268fd8154-kube-api-access-qrl9m\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.664432 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.664292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c6d6c9-1832-4e87-9cc9-c28268fd8154-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.666806 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.666776    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c6d6c9-1832-4e87-9cc9-c28268fd8154-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.673055 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.673031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrl9m\" (UniqueName: \"kubernetes.io/projected/53c6d6c9-1832-4e87-9cc9-c28268fd8154-kube-api-access-qrl9m\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.756359 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.756327    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:43.816146 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:43.815568    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"]
Sep 17 23:50:43.818871 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:50:43.818833    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ce118e_5671_4647_adcd_72c9700ce69a.slice/crio-8fcbfcfb694704e6935a9f4660a548a80ce6b41629e3dc21c9b12d2734f26779 WatchSource:0}: Error finding container 8fcbfcfb694704e6935a9f4660a548a80ce6b41629e3dc21c9b12d2734f26779: Status 404 returned error can't find the container with id 8fcbfcfb694704e6935a9f4660a548a80ce6b41629e3dc21c9b12d2734f26779
Sep 17 23:50:44.039348 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:44.039316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"]
Sep 17 23:50:44.041932 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:50:44.041898    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c6d6c9_1832_4e87_9cc9_c28268fd8154.slice/crio-4b6f99b3e20d31e45d3d919c3e8581c4b5658ba4f26530913b84aff3d8d95c63 WatchSource:0}: Error finding container 4b6f99b3e20d31e45d3d919c3e8581c4b5658ba4f26530913b84aff3d8d95c63: Status 404 returned error can't find the container with id 4b6f99b3e20d31e45d3d919c3e8581c4b5658ba4f26530913b84aff3d8d95c63
Sep 17 23:50:44.327712 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:44.327612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" event={"ID":"53c6d6c9-1832-4e87-9cc9-c28268fd8154","Type":"ContainerStarted","Data":"38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2"}
Sep 17 23:50:44.327712 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:44.327672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" event={"ID":"53c6d6c9-1832-4e87-9cc9-c28268fd8154","Type":"ContainerStarted","Data":"4b6f99b3e20d31e45d3d919c3e8581c4b5658ba4f26530913b84aff3d8d95c63"}
Sep 17 23:50:44.328198 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:44.327727    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:50:44.329180 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:44.329158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" event={"ID":"71ce118e-5671-4647-adcd-72c9700ce69a","Type":"ContainerStarted","Data":"c6118ef6ec019dd84bd32d5baef5f3eb849ffa6c19c475df716d80f57eaae66a"}
Sep 17 23:50:44.329290 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:44.329193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" event={"ID":"71ce118e-5671-4647-adcd-72c9700ce69a","Type":"ContainerStarted","Data":"8fcbfcfb694704e6935a9f4660a548a80ce6b41629e3dc21c9b12d2734f26779"}
Sep 17 23:50:44.349050 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:44.349000    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" podStartSLOduration=1.348982693 podStartE2EDuration="1.348982693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:50:44.347549324 +0000 UTC m=+2404.799380815" watchObservedRunningTime="2026-09-17 23:50:44.348982693 +0000 UTC m=+2404.800814182"
Sep 17 23:50:47.343884 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:47.343848    2566 generic.go:356] "Generic (PLEG): container finished" podID="71ce118e-5671-4647-adcd-72c9700ce69a" containerID="c6118ef6ec019dd84bd32d5baef5f3eb849ffa6c19c475df716d80f57eaae66a" exitCode=0
Sep 17 23:50:47.344250 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:47.343893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" event={"ID":"71ce118e-5671-4647-adcd-72c9700ce69a","Type":"ContainerDied","Data":"c6118ef6ec019dd84bd32d5baef5f3eb849ffa6c19c475df716d80f57eaae66a"}
Sep 17 23:50:48.351338 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:48.351303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" event={"ID":"71ce118e-5671-4647-adcd-72c9700ce69a","Type":"ContainerStarted","Data":"6f285b29bb9f9afd4cdb9366900978f92103dae056414bc545207febf333bdb7"}
Sep 17 23:50:48.375069 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:48.375021    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podStartSLOduration=2.491683824 podStartE2EDuration="5.3750061s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.883322276s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:50:48.372302569 +0000 UTC m=+2408.824134106" watchObservedRunningTime="2026-09-17 23:50:48.3750061 +0000 UTC m=+2408.826837578"
Sep 17 23:50:48.532809 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:48.532782    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:53.532083 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:53.532047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:50:53.533380 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:50:53.533353    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:51:03.532506 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:51:03.532419    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:51:13.533097 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:51:13.533053    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:51:15.335251 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:51:15.335222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:51:23.532976 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:51:23.532935    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:51:33.532804 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:51:33.532763    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:51:43.532264 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:51:43.532216    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:51:53.532647 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:51:53.532612    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:52:03.542644 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:03.542600    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:52:03.550952 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:03.550924    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:52:22.264325 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:22.263976    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"]
Sep 17 23:52:22.268078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:22.268051    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"]
Sep 17 23:52:25.335863 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:25.335817    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:52:35.335403 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:35.335304    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:52:37.287890 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:37.287840    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main" containerID="cri-o://38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2" gracePeriod=45
Sep 17 23:52:37.302948 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:37.302910    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" containerID="cri-o://6f285b29bb9f9afd4cdb9366900978f92103dae056414bc545207febf333bdb7" gracePeriod=45
Sep 17 23:52:37.543934 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:37.543852    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 23:52:37.893218 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:37.893187    2566 generic.go:356] "Generic (PLEG): container finished" podID="71ce118e-5671-4647-adcd-72c9700ce69a" containerID="6f285b29bb9f9afd4cdb9366900978f92103dae056414bc545207febf333bdb7" exitCode=0
Sep 17 23:52:37.893376 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:37.893256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" event={"ID":"71ce118e-5671-4647-adcd-72c9700ce69a","Type":"ContainerDied","Data":"6f285b29bb9f9afd4cdb9366900978f92103dae056414bc545207febf333bdb7"}
Sep 17 23:52:37.948224 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:37.948203    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:52:38.007099 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007075    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-kserve-provision-location\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007131    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-dshm\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ce118e-5671-4647-adcd-72c9700ce69a-tls-certs\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007188    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-tmp-dir\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007227 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007218    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-home\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007504 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z55c7\" (UniqueName: \"kubernetes.io/projected/71ce118e-5671-4647-adcd-72c9700ce69a-kube-api-access-z55c7\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007504 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007286    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ce118e-5671-4647-adcd-72c9700ce69a-cabundle-cert\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007504 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007322    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-model-cache\") pod \"71ce118e-5671-4647-adcd-72c9700ce69a\" (UID: \"71ce118e-5671-4647-adcd-72c9700ce69a\") "
Sep 17 23:52:38.007795 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007633    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:52:38.007795 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007753    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ce118e-5671-4647-adcd-72c9700ce69a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:52:38.007795 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-model-cache" (OuterVolumeSpecName: "model-cache") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:52:38.007956 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.007910    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-home" (OuterVolumeSpecName: "home") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:52:38.009751 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.009721    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-dshm" (OuterVolumeSpecName: "dshm") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:52:38.009928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.009899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ce118e-5671-4647-adcd-72c9700ce69a-kube-api-access-z55c7" (OuterVolumeSpecName: "kube-api-access-z55c7") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "kube-api-access-z55c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:52:38.009928 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.009905    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ce118e-5671-4647-adcd-72c9700ce69a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:52:38.038247 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.038220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71ce118e-5671-4647-adcd-72c9700ce69a" (UID: "71ce118e-5671-4647-adcd-72c9700ce69a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:52:38.108852 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108792    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.108852 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108816    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ce118e-5671-4647-adcd-72c9700ce69a-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.108995 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108855    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.108995 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108867    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.108995 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108879    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z55c7\" (UniqueName: \"kubernetes.io/projected/71ce118e-5671-4647-adcd-72c9700ce69a-kube-api-access-z55c7\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.108995 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108891    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ce118e-5671-4647-adcd-72c9700ce69a-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.108995 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108903    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.108995 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.108915    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ce118e-5671-4647-adcd-72c9700ce69a-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:52:38.900759 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.900721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt" event={"ID":"71ce118e-5671-4647-adcd-72c9700ce69a","Type":"ContainerDied","Data":"8fcbfcfb694704e6935a9f4660a548a80ce6b41629e3dc21c9b12d2734f26779"}
Sep 17 23:52:38.901259 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.900785    2566 scope.go:117] "RemoveContainer" containerID="6f285b29bb9f9afd4cdb9366900978f92103dae056414bc545207febf333bdb7"
Sep 17 23:52:38.901259 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.900739    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"
Sep 17 23:52:38.910777 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.910759    2566 scope.go:117] "RemoveContainer" containerID="c6118ef6ec019dd84bd32d5baef5f3eb849ffa6c19c475df716d80f57eaae66a"
Sep 17 23:52:38.925180 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.925155    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"]
Sep 17 23:52:38.927873 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:38.927845    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5646c7cf78-j2wmt"]
Sep 17 23:52:40.015136 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:40.015101    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" path="/var/lib/kubelet/pods/71ce118e-5671-4647-adcd-72c9700ce69a/volumes"
Sep 17 23:52:45.335567 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:45.335514    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:52:45.336032 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:45.335659    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:52:55.335516 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:52:55.335466    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:53:05.335743 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:05.335695    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:53:07.531284 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:07.531256    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:53:07.634944 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:07.634881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c6d6c9-1832-4e87-9cc9-c28268fd8154-tls-certs\") pod \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") "
Sep 17 23:53:07.634944 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:07.634932    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrl9m\" (UniqueName: \"kubernetes.io/projected/53c6d6c9-1832-4e87-9cc9-c28268fd8154-kube-api-access-qrl9m\") pod \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\" (UID: \"53c6d6c9-1832-4e87-9cc9-c28268fd8154\") "
Sep 17 23:53:07.637015 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:07.636988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c6d6c9-1832-4e87-9cc9-c28268fd8154-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53c6d6c9-1832-4e87-9cc9-c28268fd8154" (UID: "53c6d6c9-1832-4e87-9cc9-c28268fd8154"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:53:07.637130 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:07.637089    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c6d6c9-1832-4e87-9cc9-c28268fd8154-kube-api-access-qrl9m" (OuterVolumeSpecName: "kube-api-access-qrl9m") pod "53c6d6c9-1832-4e87-9cc9-c28268fd8154" (UID: "53c6d6c9-1832-4e87-9cc9-c28268fd8154"). InnerVolumeSpecName "kube-api-access-qrl9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:53:07.735803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:07.735777    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrl9m\" (UniqueName: \"kubernetes.io/projected/53c6d6c9-1832-4e87-9cc9-c28268fd8154-kube-api-access-qrl9m\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:53:07.735803 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:07.735799    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c6d6c9-1832-4e87-9cc9-c28268fd8154-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:53:08.044251 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.044221    2566 generic.go:356] "Generic (PLEG): container finished" podID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerID="38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2" exitCode=0
Sep 17 23:53:08.044370 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.044263    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"
Sep 17 23:53:08.044370 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.044307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" event={"ID":"53c6d6c9-1832-4e87-9cc9-c28268fd8154","Type":"ContainerDied","Data":"38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2"}
Sep 17 23:53:08.044370 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.044356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk" event={"ID":"53c6d6c9-1832-4e87-9cc9-c28268fd8154","Type":"ContainerDied","Data":"4b6f99b3e20d31e45d3d919c3e8581c4b5658ba4f26530913b84aff3d8d95c63"}
Sep 17 23:53:08.044533 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.044386    2566 scope.go:117] "RemoveContainer" containerID="38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2"
Sep 17 23:53:08.052504 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.052486    2566 scope.go:117] "RemoveContainer" containerID="38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2"
Sep 17 23:53:08.052800 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:53:08.052774    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2\": container with ID starting with 38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2 not found: ID does not exist" containerID="38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2"
Sep 17 23:53:08.052895 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.052805    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2"} err="failed to get container status \"38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2\": rpc error: code = NotFound desc = could not find container \"38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2\": container with ID starting with 38611ecd901dbc3d7eb5cd5c12737f69c365729ab716302220bfafc73a8929d2 not found: ID does not exist"
Sep 17 23:53:08.060880 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.060859    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"]
Sep 17 23:53:08.066442 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:08.066423    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6h7jk"]
Sep 17 23:53:10.014433 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:10.014400    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" path="/var/lib/kubelet/pods/53c6d6c9-1832-4e87-9cc9-c28268fd8154/volumes"
Sep 17 23:53:27.196068 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196036    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"]
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196391    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main"
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196404    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main"
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196413    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main"
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196419    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main"
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196431    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="storage-initializer"
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196437    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="storage-initializer"
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196496    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ce118e-5671-4647-adcd-72c9700ce69a" containerName="main"
Sep 17 23:53:27.196562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.196504    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c6d6c9-1832-4e87-9cc9-c28268fd8154" containerName="main"
Sep 17 23:53:27.198602 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.198562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.202078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.201825    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 23:53:27.202078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.201920    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 23:53:27.202078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.201920    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 23:53:27.202078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.201922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-l7n9m\""
Sep 17 23:53:27.202078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.201998    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:53:27.202078 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.202029    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 23:53:27.210455 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.210428    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"]
Sep 17 23:53:27.278429 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-cabundle-cert\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.278556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tmp-dir\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.278556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tls-certs\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.278556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kserve-provision-location\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.278556 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxc7s\" (UniqueName: \"kubernetes.io/projected/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kube-api-access-dxc7s\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.278730 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-model-cache\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.278730 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-home\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.278730 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.278641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-dshm\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379267 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-home\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379267 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-dshm\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379449 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-cabundle-cert\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379449 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tmp-dir\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379616 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tls-certs\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379616 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kserve-provision-location\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379616 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxc7s\" (UniqueName: \"kubernetes.io/projected/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kube-api-access-dxc7s\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379616 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-model-cache\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379862 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-home\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379862 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tmp-dir\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.379980 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-model-cache\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.380040 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.379985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kserve-provision-location\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.380258 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.380230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-cabundle-cert\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.381675 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.381653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-dshm\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.382004 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.381987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tls-certs\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.387421 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.387397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxc7s\" (UniqueName: \"kubernetes.io/projected/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kube-api-access-dxc7s\") pod \"stop-feature-test-kserve-54c98ddcdd-p9g2s\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.452412 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.452356    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"]
Sep 17 23:53:27.455274 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.455246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.458476 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.458451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-b9tfv\""
Sep 17 23:53:27.465629 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.465608    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"]
Sep 17 23:53:27.480626 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.480603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rhkq\" (UniqueName: \"kubernetes.io/projected/57d9ff69-514a-447c-9474-8b337744509a-kube-api-access-5rhkq\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.480717 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.480641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d9ff69-514a-447c-9474-8b337744509a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.511363 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.511338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:27.581335 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.581308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhkq\" (UniqueName: \"kubernetes.io/projected/57d9ff69-514a-447c-9474-8b337744509a-kube-api-access-5rhkq\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.581441 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.581348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d9ff69-514a-447c-9474-8b337744509a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.586295 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.586268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d9ff69-514a-447c-9474-8b337744509a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.589350 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.589325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhkq\" (UniqueName: \"kubernetes.io/projected/57d9ff69-514a-447c-9474-8b337744509a-kube-api-access-5rhkq\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.766335 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.766308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:27.792168 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:27.792140    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"]
Sep 17 23:53:27.793556 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:53:27.793533    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ced1de_44f9_40f8_94a1_ab34df5a4b31.slice/crio-a0d47ace4c6c0d10b93c18e2cdac40008da32e3b7d0d4c53491ec39843eb66d2 WatchSource:0}: Error finding container a0d47ace4c6c0d10b93c18e2cdac40008da32e3b7d0d4c53491ec39843eb66d2: Status 404 returned error can't find the container with id a0d47ace4c6c0d10b93c18e2cdac40008da32e3b7d0d4c53491ec39843eb66d2
Sep 17 23:53:28.065277 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:28.065246    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"]
Sep 17 23:53:28.066680 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:53:28.066643    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d9ff69_514a_447c_9474_8b337744509a.slice/crio-86f92b3acf5a0ebf0336fef16fbfce6995d461bfdea0c4de9aff154b698e1070 WatchSource:0}: Error finding container 86f92b3acf5a0ebf0336fef16fbfce6995d461bfdea0c4de9aff154b698e1070: Status 404 returned error can't find the container with id 86f92b3acf5a0ebf0336fef16fbfce6995d461bfdea0c4de9aff154b698e1070
Sep 17 23:53:28.145192 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:28.145154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" event={"ID":"57d9ff69-514a-447c-9474-8b337744509a","Type":"ContainerStarted","Data":"688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a"}
Sep 17 23:53:28.145370 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:28.145222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" event={"ID":"57d9ff69-514a-447c-9474-8b337744509a","Type":"ContainerStarted","Data":"86f92b3acf5a0ebf0336fef16fbfce6995d461bfdea0c4de9aff154b698e1070"}
Sep 17 23:53:28.145660 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:28.145594    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:53:28.146933 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:28.146911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" event={"ID":"e3ced1de-44f9-40f8-94a1-ab34df5a4b31","Type":"ContainerStarted","Data":"0cc6cfc9a5279d1138a18be2967a013c638786dc8981a7b7f4a02594853a54fd"}
Sep 17 23:53:28.147046 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:28.146956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" event={"ID":"e3ced1de-44f9-40f8-94a1-ab34df5a4b31","Type":"ContainerStarted","Data":"a0d47ace4c6c0d10b93c18e2cdac40008da32e3b7d0d4c53491ec39843eb66d2"}
Sep 17 23:53:28.165214 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:28.165140    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" podStartSLOduration=1.165121287 podStartE2EDuration="1.165121287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:53:28.163965634 +0000 UTC m=+2568.615797123" watchObservedRunningTime="2026-09-17 23:53:28.165121287 +0000 UTC m=+2568.616952784"
Sep 17 23:53:31.174499 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:31.174462    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerID="0cc6cfc9a5279d1138a18be2967a013c638786dc8981a7b7f4a02594853a54fd" exitCode=0
Sep 17 23:53:31.174893 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:31.174502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" event={"ID":"e3ced1de-44f9-40f8-94a1-ab34df5a4b31","Type":"ContainerDied","Data":"0cc6cfc9a5279d1138a18be2967a013c638786dc8981a7b7f4a02594853a54fd"}
Sep 17 23:53:32.181273 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:32.181239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" event={"ID":"e3ced1de-44f9-40f8-94a1-ab34df5a4b31","Type":"ContainerStarted","Data":"357e28fd84e78af16cca7c18287d4dac9e9ea0a67c6f6e3b99b11af916c540ad"}
Sep 17 23:53:32.203805 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:32.203749    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podStartSLOduration=2.909590622 podStartE2EDuration="5.203730917s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.294140295s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:53:32.201439505 +0000 UTC m=+2572.653270992" watchObservedRunningTime="2026-09-17 23:53:32.203730917 +0000 UTC m=+2572.655562406"
Sep 17 23:53:32.512072 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:32.512042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:37.511823 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:37.511783    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:53:37.513415 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:37.513388    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:53:47.512280 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:47.512235    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:53:57.512009 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:57.511970    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:53:59.164969 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:53:59.164940    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:54:07.512230 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:54:07.512132    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:54:17.512538 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:54:17.512491    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:54:27.511835 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:54:27.511789    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:54:37.512668 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:54:37.512629    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:54:47.512638 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:54:47.512597    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:54:57.511758 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:54:57.511715    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:55:07.512550 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:07.512502    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:55:17.521890 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:17.521859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:55:17.529562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:17.529534    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:55:19.211047 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:19.211016    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"]
Sep 17 23:55:19.459461 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:19.459428    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"]
Sep 17 23:55:29.165246 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:29.165192    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:55:34.234645 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:34.234529    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" containerID="cri-o://357e28fd84e78af16cca7c18287d4dac9e9ea0a67c6f6e3b99b11af916c540ad" gracePeriod=45
Sep 17 23:55:34.478934 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:34.478893    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main" containerID="cri-o://688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a" gracePeriod=45
Sep 17 23:55:34.522633 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:34.522531    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 23:55:34.803119 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:34.803037    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerID="357e28fd84e78af16cca7c18287d4dac9e9ea0a67c6f6e3b99b11af916c540ad" exitCode=0
Sep 17 23:55:34.803119 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:34.803085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" event={"ID":"e3ced1de-44f9-40f8-94a1-ab34df5a4b31","Type":"ContainerDied","Data":"357e28fd84e78af16cca7c18287d4dac9e9ea0a67c6f6e3b99b11af916c540ad"}
Sep 17 23:55:34.890821 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:34.890797    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:55:35.010207 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kserve-provision-location\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.010349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010231    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxc7s\" (UniqueName: \"kubernetes.io/projected/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kube-api-access-dxc7s\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.010349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010262    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-home\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.010349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010296    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-dshm\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.010349 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010337    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tls-certs\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.010606 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010354    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tmp-dir\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.010606 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-model-cache\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.010606 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.010413    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-cabundle-cert\") pod \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\" (UID: \"e3ced1de-44f9-40f8-94a1-ab34df5a4b31\") "
Sep 17 23:55:35.011158 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.011031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:55:35.011158 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.011122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:55:35.011304 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.011163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-home" (OuterVolumeSpecName: "home") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:55:35.012810 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.012783    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kube-api-access-dxc7s" (OuterVolumeSpecName: "kube-api-access-dxc7s") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "kube-api-access-dxc7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:55:35.012917 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.012861    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:55:35.012968 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.012946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-dshm" (OuterVolumeSpecName: "dshm") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:55:35.022618 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.022588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:55:35.040730 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.040708    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3ced1de-44f9-40f8-94a1-ab34df5a4b31" (UID: "e3ced1de-44f9-40f8-94a1-ab34df5a4b31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:55:35.111386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111327    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.111386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111350    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.111386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111360    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.111386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111369    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.111386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111377    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.111386 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111386    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxc7s\" (UniqueName: \"kubernetes.io/projected/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-kube-api-access-dxc7s\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.111669 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111394    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.111669 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.111402    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ced1de-44f9-40f8-94a1-ab34df5a4b31-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:55:35.818913 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.818871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s" event={"ID":"e3ced1de-44f9-40f8-94a1-ab34df5a4b31","Type":"ContainerDied","Data":"a0d47ace4c6c0d10b93c18e2cdac40008da32e3b7d0d4c53491ec39843eb66d2"}
Sep 17 23:55:35.818913 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.818892    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"
Sep 17 23:55:35.819526 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.818967    2566 scope.go:117] "RemoveContainer" containerID="357e28fd84e78af16cca7c18287d4dac9e9ea0a67c6f6e3b99b11af916c540ad"
Sep 17 23:55:35.829153 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.829133    2566 scope.go:117] "RemoveContainer" containerID="0cc6cfc9a5279d1138a18be2967a013c638786dc8981a7b7f4a02594853a54fd"
Sep 17 23:55:35.842571 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.842544    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"]
Sep 17 23:55:35.848743 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:35.848719    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-p9g2s"]
Sep 17 23:55:36.014547 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:36.014515    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" path="/var/lib/kubelet/pods/e3ced1de-44f9-40f8-94a1-ab34df5a4b31/volumes"
Sep 17 23:55:37.133382 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.133347    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"]
Sep 17 23:55:37.133935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.133850    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="storage-initializer"
Sep 17 23:55:37.133935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.133871    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="storage-initializer"
Sep 17 23:55:37.133935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.133910    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main"
Sep 17 23:55:37.133935 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.133920    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main"
Sep 17 23:55:37.134162 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.134008    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ced1de-44f9-40f8-94a1-ab34df5a4b31" containerName="main"
Sep 17 23:55:37.137433 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.137411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.140002 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.139978    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-95pgw\""
Sep 17 23:55:37.144865 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.144840    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"]
Sep 17 23:55:37.228805 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.228780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8b6p\" (UniqueName: \"kubernetes.io/projected/00e5714c-98fe-4e65-afa6-9114a436720f-kube-api-access-w8b6p\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.228941 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.228841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e5714c-98fe-4e65-afa6-9114a436720f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.329326 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.329301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e5714c-98fe-4e65-afa6-9114a436720f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.329432 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.329360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8b6p\" (UniqueName: \"kubernetes.io/projected/00e5714c-98fe-4e65-afa6-9114a436720f-kube-api-access-w8b6p\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.331683 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.331665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e5714c-98fe-4e65-afa6-9114a436720f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.337618 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.337596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8b6p\" (UniqueName: \"kubernetes.io/projected/00e5714c-98fe-4e65-afa6-9114a436720f-kube-api-access-w8b6p\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.450615 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.450537    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.721900 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.721876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"]
Sep 17 23:55:37.724202 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:55:37.724179    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e5714c_98fe_4e65_afa6_9114a436720f.slice/crio-b681cf21b7a12a51e15eae3a04242c3155c29d7087b7bb12e5e1dbcb2a1be200 WatchSource:0}: Error finding container b681cf21b7a12a51e15eae3a04242c3155c29d7087b7bb12e5e1dbcb2a1be200: Status 404 returned error can't find the container with id b681cf21b7a12a51e15eae3a04242c3155c29d7087b7bb12e5e1dbcb2a1be200
Sep 17 23:55:37.832068 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.832032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" event={"ID":"00e5714c-98fe-4e65-afa6-9114a436720f","Type":"ContainerStarted","Data":"6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2"}
Sep 17 23:55:37.832235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.832095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" event={"ID":"00e5714c-98fe-4e65-afa6-9114a436720f","Type":"ContainerStarted","Data":"b681cf21b7a12a51e15eae3a04242c3155c29d7087b7bb12e5e1dbcb2a1be200"}
Sep 17 23:55:37.832235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.832219    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:55:37.866319 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:37.866260    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" podStartSLOduration=0.86624465 podStartE2EDuration="866.24465ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:55:37.864385666 +0000 UTC m=+2698.316217155" watchObservedRunningTime="2026-09-17 23:55:37.86624465 +0000 UTC m=+2698.318076139"
Sep 17 23:55:39.164826 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:39.164786    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:55:40.226825 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:40.226797    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:55:40.234993 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:40.234965    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:55:40.236124 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:40.236097    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 17 23:55:40.243321 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:40.243304    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 17 23:55:49.164983 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:49.164941    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:55:49.165438 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:49.165060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:55:59.165462 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:55:59.165418    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:56:04.729144 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.729119    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:56:04.832193 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.832129    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rhkq\" (UniqueName: \"kubernetes.io/projected/57d9ff69-514a-447c-9474-8b337744509a-kube-api-access-5rhkq\") pod \"57d9ff69-514a-447c-9474-8b337744509a\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") "
Sep 17 23:56:04.832193 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.832184    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d9ff69-514a-447c-9474-8b337744509a-tls-certs\") pod \"57d9ff69-514a-447c-9474-8b337744509a\" (UID: \"57d9ff69-514a-447c-9474-8b337744509a\") "
Sep 17 23:56:04.834164 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.834130    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d9ff69-514a-447c-9474-8b337744509a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57d9ff69-514a-447c-9474-8b337744509a" (UID: "57d9ff69-514a-447c-9474-8b337744509a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:56:04.834275 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.834183    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d9ff69-514a-447c-9474-8b337744509a-kube-api-access-5rhkq" (OuterVolumeSpecName: "kube-api-access-5rhkq") pod "57d9ff69-514a-447c-9474-8b337744509a" (UID: "57d9ff69-514a-447c-9474-8b337744509a"). InnerVolumeSpecName "kube-api-access-5rhkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:56:04.933440 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.933408    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d9ff69-514a-447c-9474-8b337744509a-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:56:04.933440 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.933438    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rhkq\" (UniqueName: \"kubernetes.io/projected/57d9ff69-514a-447c-9474-8b337744509a-kube-api-access-5rhkq\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:56:04.972368 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.972336    2566 generic.go:356] "Generic (PLEG): container finished" podID="57d9ff69-514a-447c-9474-8b337744509a" containerID="688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a" exitCode=0
Sep 17 23:56:04.972474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.972390    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"
Sep 17 23:56:04.972474 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.972422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" event={"ID":"57d9ff69-514a-447c-9474-8b337744509a","Type":"ContainerDied","Data":"688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a"}
Sep 17 23:56:04.972561 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.972503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87" event={"ID":"57d9ff69-514a-447c-9474-8b337744509a","Type":"ContainerDied","Data":"86f92b3acf5a0ebf0336fef16fbfce6995d461bfdea0c4de9aff154b698e1070"}
Sep 17 23:56:04.972561 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.972529    2566 scope.go:117] "RemoveContainer" containerID="688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a"
Sep 17 23:56:04.981610 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.981569    2566 scope.go:117] "RemoveContainer" containerID="688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a"
Sep 17 23:56:04.981894 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:56:04.981875    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a\": container with ID starting with 688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a not found: ID does not exist" containerID="688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a"
Sep 17 23:56:04.981995 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.981904    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a"} err="failed to get container status \"688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a\": rpc error: code = NotFound desc = could not find container \"688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a\": container with ID starting with 688d98c6fc9edd5a7d3a6473d567993521b4cef323e4bdeb3ef92d51d6ed115a not found: ID does not exist"
Sep 17 23:56:04.992774 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.992747    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"]
Sep 17 23:56:04.998206 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:04.998186    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s7z87"]
Sep 17 23:56:06.013641 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:06.013597    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57d9ff69-514a-447c-9474-8b337744509a" path="/var/lib/kubelet/pods/57d9ff69-514a-447c-9474-8b337744509a/volumes"
Sep 17 23:56:08.838695 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:56:08.838663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:57:38.838618 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:57:38.838553    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:57:38.850077 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:57:38.850046    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"]
Sep 17 23:57:48.838962 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:57:48.838912    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:57:53.872623 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:57:53.872548    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main" containerID="cri-o://6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2" gracePeriod=45
Sep 17 23:57:58.839012 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:57:58.838969    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:57:58.839408 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:57:58.839082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:58:08.838849 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:08.838800    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:58:18.839514 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:18.839462    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:58:24.124694 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.124669    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:58:24.192562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.192502    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e5714c-98fe-4e65-afa6-9114a436720f-tls-certs\") pod \"00e5714c-98fe-4e65-afa6-9114a436720f\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") "
Sep 17 23:58:24.192711 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.192597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8b6p\" (UniqueName: \"kubernetes.io/projected/00e5714c-98fe-4e65-afa6-9114a436720f-kube-api-access-w8b6p\") pod \"00e5714c-98fe-4e65-afa6-9114a436720f\" (UID: \"00e5714c-98fe-4e65-afa6-9114a436720f\") "
Sep 17 23:58:24.194525 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.194503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e5714c-98fe-4e65-afa6-9114a436720f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00e5714c-98fe-4e65-afa6-9114a436720f" (UID: "00e5714c-98fe-4e65-afa6-9114a436720f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:58:24.194646 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.194625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e5714c-98fe-4e65-afa6-9114a436720f-kube-api-access-w8b6p" (OuterVolumeSpecName: "kube-api-access-w8b6p") pod "00e5714c-98fe-4e65-afa6-9114a436720f" (UID: "00e5714c-98fe-4e65-afa6-9114a436720f"). InnerVolumeSpecName "kube-api-access-w8b6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:58:24.293238 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.293208    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e5714c-98fe-4e65-afa6-9114a436720f-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:58:24.293238 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.293233    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8b6p\" (UniqueName: \"kubernetes.io/projected/00e5714c-98fe-4e65-afa6-9114a436720f-kube-api-access-w8b6p\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 17 23:58:24.654531 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.654498    2566 generic.go:356] "Generic (PLEG): container finished" podID="00e5714c-98fe-4e65-afa6-9114a436720f" containerID="6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2" exitCode=0
Sep 17 23:58:24.654703 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.654547    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"
Sep 17 23:58:24.654703 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.654605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" event={"ID":"00e5714c-98fe-4e65-afa6-9114a436720f","Type":"ContainerDied","Data":"6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2"}
Sep 17 23:58:24.654703 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.654652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k" event={"ID":"00e5714c-98fe-4e65-afa6-9114a436720f","Type":"ContainerDied","Data":"b681cf21b7a12a51e15eae3a04242c3155c29d7087b7bb12e5e1dbcb2a1be200"}
Sep 17 23:58:24.654703 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.654675    2566 scope.go:117] "RemoveContainer" containerID="6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2"
Sep 17 23:58:24.665011 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.664984    2566 scope.go:117] "RemoveContainer" containerID="6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2"
Sep 17 23:58:24.665318 ip-10-0-134-66 kubenswrapper[2566]: E0917 23:58:24.665279    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2\": container with ID starting with 6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2 not found: ID does not exist" containerID="6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2"
Sep 17 23:58:24.665399 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.665326    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2"} err="failed to get container status \"6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2\": rpc error: code = NotFound desc = could not find container \"6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2\": container with ID starting with 6d53f0f3b4fdcf5226c8524a179314bb9a6f36fa7a61918cdc003700e8b2dcc2 not found: ID does not exist"
Sep 17 23:58:24.676664 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.676638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"]
Sep 17 23:58:24.678843 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:24.678820    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7hv2k"]
Sep 17 23:58:26.013562 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:26.013522    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" path="/var/lib/kubelet/pods/00e5714c-98fe-4e65-afa6-9114a436720f/volumes"
Sep 17 23:58:32.819529 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.819467    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"]
Sep 17 23:58:32.820048 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.819983    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main"
Sep 17 23:58:32.820048 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.820005    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main"
Sep 17 23:58:32.820048 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.820016    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main"
Sep 17 23:58:32.820048 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.820024    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main"
Sep 17 23:58:32.820274 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.820142    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e5714c-98fe-4e65-afa6-9114a436720f" containerName="main"
Sep 17 23:58:32.820274 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.820158    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d9ff69-514a-447c-9474-8b337744509a" containerName="main"
Sep 17 23:58:32.825013 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.824989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.827852 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.827828    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-jj72z\""
Sep 17 23:58:32.827980 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.827882    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 23:58:32.827980 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.827923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 23:58:32.827980 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.827946    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:58:32.828158 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.828033    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 23:58:32.828158 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.828151    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 23:58:32.834863 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.834737    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"]
Sep 17 23:58:32.954071 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.954071 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-dshm\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.954219 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tmp-dir\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.954219 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tls-certs\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.954219 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frdh\" (UniqueName: \"kubernetes.io/projected/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kube-api-access-8frdh\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.954219 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-home\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.954375 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-model-cache\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:32.954375 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:32.954252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055193 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-home\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055314 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-model-cache\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055314 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-dshm\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tmp-dir\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tls-certs\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055439 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frdh\" (UniqueName: \"kubernetes.io/projected/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kube-api-access-8frdh\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-home\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-model-cache\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.055760 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.055741    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.056056 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.056035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.056136 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.056083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tmp-dir\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.057841 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.057818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-dshm\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.057936 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.057900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tls-certs\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.068088 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.068063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frdh\" (UniqueName: \"kubernetes.io/projected/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kube-api-access-8frdh\") pod \"lora-single-adapter-test-kserve-6f7545d58b-ntn5r\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.138596 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.138521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:33.440814 ip-10-0-134-66 kubenswrapper[2566]: W0917 23:58:33.440775    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e314e3d_a66e_4f91_9ef7_17a15f5a3eee.slice/crio-f3fc399e3597b0e57ffb52c0a570ba059cdf48c2626b480a6920b7a05861a407 WatchSource:0}: Error finding container f3fc399e3597b0e57ffb52c0a570ba059cdf48c2626b480a6920b7a05861a407: Status 404 returned error can't find the container with id f3fc399e3597b0e57ffb52c0a570ba059cdf48c2626b480a6920b7a05861a407
Sep 17 23:58:33.441054 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.441028    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"]
Sep 17 23:58:33.711475 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.711391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" event={"ID":"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee","Type":"ContainerStarted","Data":"7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df"}
Sep 17 23:58:33.711672 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:33.711489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" event={"ID":"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee","Type":"ContainerStarted","Data":"f3fc399e3597b0e57ffb52c0a570ba059cdf48c2626b480a6920b7a05861a407"}
Sep 17 23:58:36.728210 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:36.728169    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerID="7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df" exitCode=0
Sep 17 23:58:36.728650 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:36.728246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" event={"ID":"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee","Type":"ContainerDied","Data":"7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df"}
Sep 17 23:58:37.735714 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:37.735677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" event={"ID":"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee","Type":"ContainerStarted","Data":"c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5"}
Sep 17 23:58:37.757772 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:37.757723    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podStartSLOduration=3.064333364 podStartE2EDuration="5.757707s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.693373636s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:58:37.75539373 +0000 UTC m=+2878.207225218" watchObservedRunningTime="2026-09-17 23:58:37.757707 +0000 UTC m=+2878.209538487"
Sep 17 23:58:38.139413 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:38.139326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:43.139382 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:43.139350    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:58:43.141148 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:43.141109    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 23:58:53.139787 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:58:53.139741    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 23:59:03.140019 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:59:03.139972    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 23:59:13.139802 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:59:13.139753    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 23:59:23.139780 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:59:23.139738    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 23:59:33.139769 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:59:33.139721    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 23:59:43.139756 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:59:43.139709    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 23:59:53.149300 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:59:53.149263    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 17 23:59:53.157235 ip-10-0-134-66 kubenswrapper[2566]: I0917 23:59:53.157204    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 18 00:00:17.230723 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:17.230643    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-6f7545d58b-ntn5r_0e314e3d-a66e-4f91-9ef7-17a15f5a3eee/storage-initializer/0.log"
Sep 18 00:00:17.241135 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:17.241112    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-6f7545d58b-ntn5r_0e314e3d-a66e-4f91-9ef7-17a15f5a3eee/main/0.log"
Sep 18 00:00:18.393013 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:18.392979    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"]
Sep 18 00:00:33.416079 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:33.416011    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main" containerID="cri-o://c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5" gracePeriod=45
Sep 18 00:00:34.054200 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.054175    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 18 00:00:34.129182 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129114    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-model-cache\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129182 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129153    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tls-certs\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129182 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tmp-dir\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129423 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129200    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8frdh\" (UniqueName: \"kubernetes.io/projected/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kube-api-access-8frdh\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129423 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129249    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-home\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129423 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-dshm\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129423 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129315    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kserve-provision-location\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129423 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-cabundle-cert\") pod \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\" (UID: \"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee\") "
Sep 18 00:00:34.129716 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129416    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:00:34.129779 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-home" (OuterVolumeSpecName: "home") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:00:34.130243 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129934    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.130243 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129956    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.130243 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.129988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:00:34.131605 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.131557    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-dshm" (OuterVolumeSpecName: "dshm") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:00:34.131720 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.131658    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kube-api-access-8frdh" (OuterVolumeSpecName: "kube-api-access-8frdh") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "kube-api-access-8frdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:00:34.131777 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.131748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:00:34.141007 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.140981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:00:34.154104 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.154081    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" (UID: "0e314e3d-a66e-4f91-9ef7-17a15f5a3eee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:00:34.231033 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.231010    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.231033 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.231030    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.231158 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.231039    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frdh\" (UniqueName: \"kubernetes.io/projected/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kube-api-access-8frdh\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.231158 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.231048    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.231158 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.231056    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.231158 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.231065    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:00:34.312173 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.312134    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerID="c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5" exitCode=0
Sep 18 00:00:34.312268 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.312194    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"
Sep 18 00:00:34.312268 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.312196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" event={"ID":"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee","Type":"ContainerDied","Data":"c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5"}
Sep 18 00:00:34.312268 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.312251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r" event={"ID":"0e314e3d-a66e-4f91-9ef7-17a15f5a3eee","Type":"ContainerDied","Data":"f3fc399e3597b0e57ffb52c0a570ba059cdf48c2626b480a6920b7a05861a407"}
Sep 18 00:00:34.312389 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.312278    2566 scope.go:117] "RemoveContainer" containerID="c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5"
Sep 18 00:00:34.321919 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.321903    2566 scope.go:117] "RemoveContainer" containerID="7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df"
Sep 18 00:00:34.330033 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.330012    2566 scope.go:117] "RemoveContainer" containerID="c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5"
Sep 18 00:00:34.330314 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:00:34.330293    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5\": container with ID starting with c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5 not found: ID does not exist" containerID="c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5"
Sep 18 00:00:34.330359 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.330331    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5"} err="failed to get container status \"c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5\": rpc error: code = NotFound desc = could not find container \"c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5\": container with ID starting with c841f160a442b1e43a865f209f9d45d9e98405ad02a25a1dcb525f4fce6dc3d5 not found: ID does not exist"
Sep 18 00:00:34.330359 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.330351    2566 scope.go:117] "RemoveContainer" containerID="7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df"
Sep 18 00:00:34.330590 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:00:34.330562    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df\": container with ID starting with 7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df not found: ID does not exist" containerID="7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df"
Sep 18 00:00:34.330642 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.330596    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df"} err="failed to get container status \"7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df\": rpc error: code = NotFound desc = could not find container \"7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df\": container with ID starting with 7f55a1adf068ef96dd088d4f9c0359319ab1ef84f89b9a851e48324f998448df not found: ID does not exist"
Sep 18 00:00:34.335675 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.335655    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"]
Sep 18 00:00:34.341470 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:34.341448    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6f7545d58b-ntn5r"]
Sep 18 00:00:36.014459 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:36.014417    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" path="/var/lib/kubelet/pods/0e314e3d-a66e-4f91-9ef7-17a15f5a3eee/volumes"
Sep 18 00:00:40.256253 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:40.256223    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:00:40.264299 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:40.264276    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:00:40.265756 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:40.265737    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:00:40.273043 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:40.273024    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:00:45.913798 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.913757    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"]
Sep 18 00:00:45.914321 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.914297    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main"
Sep 18 00:00:45.914397 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.914325    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main"
Sep 18 00:00:45.914397 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.914340    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="storage-initializer"
Sep 18 00:00:45.914397 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.914349    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="storage-initializer"
Sep 18 00:00:45.914552 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.914454    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e314e3d-a66e-4f91-9ef7-17a15f5a3eee" containerName="main"
Sep 18 00:00:45.918914 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.918889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:45.921662 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.921634    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 18 00:00:45.921798 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.921634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 18 00:00:45.921798 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.921634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-mr78m\""
Sep 18 00:00:45.921798 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.921634    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:00:45.921798 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.921640    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 18 00:00:45.921798 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.921644    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 18 00:00:45.927942 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:45.927918    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"]
Sep 18 00:00:46.020157 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-home\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.020157 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.020337 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.020337 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.020337 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk8xf\" (UniqueName: \"kubernetes.io/projected/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kube-api-access-tk8xf\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.020441 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.020441 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-dshm\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.020441 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.020415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-model-cache\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121463 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk8xf\" (UniqueName: \"kubernetes.io/projected/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kube-api-access-tk8xf\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-dshm\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-model-cache\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121803 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-home\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121803 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121803 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.121961 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.121837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.122032 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.122006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-model-cache\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.122097 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.122055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-home\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.122155 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.122126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.122260 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.122230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.122421 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.122394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.123893 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.123871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-dshm\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.124178 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.124163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.130327 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.130302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk8xf\" (UniqueName: \"kubernetes.io/projected/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kube-api-access-tk8xf\") pod \"lora-multiple-adapters-test-kserve-5f8776b785-fm7sm\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.231543 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.231481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:46.508863 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:46.508725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"]
Sep 18 00:00:46.511612 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:00:46.511562    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8179c3_fbeb_4d5b_95a9_ba7c69be1b75.slice/crio-e93db32fdb1b0743890eac5cf87af8a0123b5f8789ca0af1dab690956683ca80 WatchSource:0}: Error finding container e93db32fdb1b0743890eac5cf87af8a0123b5f8789ca0af1dab690956683ca80: Status 404 returned error can't find the container with id e93db32fdb1b0743890eac5cf87af8a0123b5f8789ca0af1dab690956683ca80
Sep 18 00:00:47.379790 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:47.379747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" event={"ID":"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75","Type":"ContainerStarted","Data":"f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2"}
Sep 18 00:00:47.380201 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:47.379816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" event={"ID":"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75","Type":"ContainerStarted","Data":"e93db32fdb1b0743890eac5cf87af8a0123b5f8789ca0af1dab690956683ca80"}
Sep 18 00:00:50.400261 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:50.400219    2566 generic.go:356] "Generic (PLEG): container finished" podID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerID="f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2" exitCode=0
Sep 18 00:00:50.400688 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:50.400292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" event={"ID":"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75","Type":"ContainerDied","Data":"f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2"}
Sep 18 00:00:51.406560 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:51.406528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" event={"ID":"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75","Type":"ContainerStarted","Data":"b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55"}
Sep 18 00:00:51.432000 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:51.431948    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podStartSLOduration=3.365088419 podStartE2EDuration="6.431933809s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.06684539s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:00:51.429435657 +0000 UTC m=+3011.881267145" watchObservedRunningTime="2026-09-18 00:00:51.431933809 +0000 UTC m=+3011.883765353"
Sep 18 00:00:52.232110 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:52.232082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:56.231745 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:56.231696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:00:56.233254 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:00:56.233220    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 18 00:01:06.232500 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:01:06.232451    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 18 00:01:16.232000 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:01:16.231961    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 18 00:01:26.232068 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:01:26.232031    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 18 00:01:36.232677 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:01:36.232567    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 18 00:01:46.232795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:01:46.232689    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 18 00:01:56.232116 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:01:56.232067    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 18 00:02:06.242937 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:06.242897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:02:06.257636 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:06.257605    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:02:32.427423 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:32.427394    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5f8776b785-fm7sm_5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75/storage-initializer/0.log"
Sep 18 00:02:32.435852 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:32.435825    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5f8776b785-fm7sm_5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75/main/0.log"
Sep 18 00:02:33.290942 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:33.290906    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"]
Sep 18 00:02:48.314656 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:48.314569    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main" containerID="cri-o://b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55" gracePeriod=45
Sep 18 00:02:48.962525 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:48.962502    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:02:48.999955 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:48.999877    2566 generic.go:356] "Generic (PLEG): container finished" podID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerID="b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55" exitCode=0
Sep 18 00:02:48.999955 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:48.999931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" event={"ID":"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75","Type":"ContainerDied","Data":"b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55"}
Sep 18 00:02:48.999955 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:48.999937    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"
Sep 18 00:02:49.000182 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:48.999970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm" event={"ID":"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75","Type":"ContainerDied","Data":"e93db32fdb1b0743890eac5cf87af8a0123b5f8789ca0af1dab690956683ca80"}
Sep 18 00:02:49.000182 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:48.999993    2566 scope.go:117] "RemoveContainer" containerID="b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55"
Sep 18 00:02:49.008344 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.008317    2566 scope.go:117] "RemoveContainer" containerID="f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2"
Sep 18 00:02:49.017551 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.017526    2566 scope.go:117] "RemoveContainer" containerID="b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55"
Sep 18 00:02:49.017841 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:02:49.017823    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55\": container with ID starting with b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55 not found: ID does not exist" containerID="b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55"
Sep 18 00:02:49.017898 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.017851    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55"} err="failed to get container status \"b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55\": rpc error: code = NotFound desc = could not find container \"b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55\": container with ID starting with b2a6502f5b949ac5666d2351b2d09ea130e5fe2d2ea918ec55353149cd7d1a55 not found: ID does not exist"
Sep 18 00:02:49.017898 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.017867    2566 scope.go:117] "RemoveContainer" containerID="f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2"
Sep 18 00:02:49.018158 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:02:49.018120    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2\": container with ID starting with f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2 not found: ID does not exist" containerID="f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2"
Sep 18 00:02:49.018253 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.018147    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2"} err="failed to get container status \"f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2\": rpc error: code = NotFound desc = could not find container \"f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2\": container with ID starting with f54b7548a344d9eb68578d759ec8de018da281dd84ddba57da32223b253694e2 not found: ID does not exist"
Sep 18 00:02:49.027457 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tls-certs\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.027535 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tmp-dir\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.027629 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027613    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-home\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.027676 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027665    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk8xf\" (UniqueName: \"kubernetes.io/projected/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kube-api-access-tk8xf\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.027714 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-dshm\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.027749 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027717    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kserve-provision-location\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.027797 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-model-cache\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.027797 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.027791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-cabundle-cert\") pod \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\" (UID: \"5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75\") "
Sep 18 00:02:49.028161 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.028114    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-home" (OuterVolumeSpecName: "home") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:49.028161 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.028127    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:49.028431 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.028405    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:02:49.029826 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.029804    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:02:49.030146 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.030109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-dshm" (OuterVolumeSpecName: "dshm") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:49.030146 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.030128    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kube-api-access-tk8xf" (OuterVolumeSpecName: "kube-api-access-tk8xf") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "kube-api-access-tk8xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:02:49.040537 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.040516    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:49.054246 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.054221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" (UID: "5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:49.128652 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128627    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.128652 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128649    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.128795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128662    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.128795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128675    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.128795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128686    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.128795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128698    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk8xf\" (UniqueName: \"kubernetes.io/projected/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kube-api-access-tk8xf\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.128795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128709    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.128795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.128723    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:02:49.323655 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.323619    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"]
Sep 18 00:02:49.327284 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:49.327260    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f8776b785-fm7sm"]
Sep 18 00:02:50.014723 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:50.014691    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" path="/var/lib/kubelet/pods/5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75/volumes"
Sep 18 00:02:56.705311 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.705265    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"]
Sep 18 00:02:56.705787 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.705637    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main"
Sep 18 00:02:56.705787 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.705652    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main"
Sep 18 00:02:56.705787 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.705674    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="storage-initializer"
Sep 18 00:02:56.705787 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.705680    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="storage-initializer"
Sep 18 00:02:56.705787 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.705746    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8179c3-fbeb-4d5b-95a9-ba7c69be1b75" containerName="main"
Sep 18 00:02:56.707871 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.707850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:56.710668 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.710625    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 00:02:56.711785 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.711759    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-fm9k7\""
Sep 18 00:02:56.711900 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.711854    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 00:02:56.711900 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.711854    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 00:02:56.715412 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.715384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"]
Sep 18 00:02:56.794906 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.794877    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6hj\" (UniqueName: \"kubernetes.io/projected/4629195f-b6ed-47f0-83f5-29c217c5ef16-kube-api-access-4t6hj\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:56.895818 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.895796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6hj\" (UniqueName: \"kubernetes.io/projected/4629195f-b6ed-47f0-83f5-29c217c5ef16-kube-api-access-4t6hj\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:56.905729 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:56.905706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6hj\" (UniqueName: \"kubernetes.io/projected/4629195f-b6ed-47f0-83f5-29c217c5ef16-kube-api-access-4t6hj\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.110399 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.110368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.116133 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.116110    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.210736 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.210707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.210855 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.210808    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.213748 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.213724    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") DevicePath \"csi-b7c0b6b81660aa4dd8646aabe8f21d23b0535193fdcbef17ad859d105b3b6c57\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.435371 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.435344    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/116e18f5cf288dc8fecb0d77b30aea631f70117afbdf1ecc9da5a5d14780a735/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.448648 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.448621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"e2e-pvc-lora-download-w5lp4\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.718436 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.718373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:02:59.985999 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:02:59.985976    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"]
Sep 18 00:02:59.988816 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:02:59.988795    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4629195f_b6ed_47f0_83f5_29c217c5ef16.slice/crio-61ceec05b48a458c18ef451aa15ba33e9da01e392b71569bfd1b252cf3ad6e3b WatchSource:0}: Error finding container 61ceec05b48a458c18ef451aa15ba33e9da01e392b71569bfd1b252cf3ad6e3b: Status 404 returned error can't find the container with id 61ceec05b48a458c18ef451aa15ba33e9da01e392b71569bfd1b252cf3ad6e3b
Sep 18 00:03:00.053417 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:00.053392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4" event={"ID":"4629195f-b6ed-47f0-83f5-29c217c5ef16","Type":"ContainerStarted","Data":"00526560d7ff95949edb52638ead9b064bc314944754eab568efd845ed0b8226"}
Sep 18 00:03:00.053516 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:00.053427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4" event={"ID":"4629195f-b6ed-47f0-83f5-29c217c5ef16","Type":"ContainerStarted","Data":"61ceec05b48a458c18ef451aa15ba33e9da01e392b71569bfd1b252cf3ad6e3b"}
Sep 18 00:03:00.068359 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:00.068319    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4" podStartSLOduration=7.068305288 podStartE2EDuration="7.068305288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:03:00.067331466 +0000 UTC m=+3140.519162954" watchObservedRunningTime="2026-09-18 00:03:00.068305288 +0000 UTC m=+3140.520136776"
Sep 18 00:03:02.068198 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:02.068160    2566 generic.go:356] "Generic (PLEG): container finished" podID="4629195f-b6ed-47f0-83f5-29c217c5ef16" containerID="00526560d7ff95949edb52638ead9b064bc314944754eab568efd845ed0b8226" exitCode=0
Sep 18 00:03:02.068648 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:02.068201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4" event={"ID":"4629195f-b6ed-47f0-83f5-29c217c5ef16","Type":"ContainerDied","Data":"00526560d7ff95949edb52638ead9b064bc314944754eab568efd845ed0b8226"}
Sep 18 00:03:03.203439 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:03.203415    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:03:03.344868 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:03.344807    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"4629195f-b6ed-47f0-83f5-29c217c5ef16\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") "
Sep 18 00:03:03.344868 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:03.344849    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t6hj\" (UniqueName: \"kubernetes.io/projected/4629195f-b6ed-47f0-83f5-29c217c5ef16-kube-api-access-4t6hj\") pod \"4629195f-b6ed-47f0-83f5-29c217c5ef16\" (UID: \"4629195f-b6ed-47f0-83f5-29c217c5ef16\") "
Sep 18 00:03:03.347173 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:03.347148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4629195f-b6ed-47f0-83f5-29c217c5ef16-kube-api-access-4t6hj" (OuterVolumeSpecName: "kube-api-access-4t6hj") pod "4629195f-b6ed-47f0-83f5-29c217c5ef16" (UID: "4629195f-b6ed-47f0-83f5-29c217c5ef16"). InnerVolumeSpecName "kube-api-access-4t6hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:03:03.348755 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:03.348733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47" (OuterVolumeSpecName: "model-storage") pod "4629195f-b6ed-47f0-83f5-29c217c5ef16" (UID: "4629195f-b6ed-47f0-83f5-29c217c5ef16"). InnerVolumeSpecName "pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 00:03:03.445387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:03.445360    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") on node \"ip-10-0-134-66.ec2.internal\" "
Sep 18 00:03:03.445387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:03.445383    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6hj\" (UniqueName: \"kubernetes.io/projected/4629195f-b6ed-47f0-83f5-29c217c5ef16-kube-api-access-4t6hj\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:03:04.078429 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:04.078404    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"
Sep 18 00:03:04.078614 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:04.078397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4" event={"ID":"4629195f-b6ed-47f0-83f5-29c217c5ef16","Type":"ContainerDied","Data":"61ceec05b48a458c18ef451aa15ba33e9da01e392b71569bfd1b252cf3ad6e3b"}
Sep 18 00:03:04.078614 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:04.078504    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ceec05b48a458c18ef451aa15ba33e9da01e392b71569bfd1b252cf3ad6e3b"
Sep 18 00:03:04.413262 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:04.413243    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47") on node "ip-10-0-134-66.ec2.internal"
Sep 18 00:03:04.452172 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:04.452149    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"csi-b7c0b6b81660aa4dd8646aabe8f21d23b0535193fdcbef17ad859d105b3b6c57\""
Sep 18 00:03:50.851007 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.850968    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"]
Sep 18 00:03:50.851545 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.851357    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4629195f-b6ed-47f0-83f5-29c217c5ef16" containerName="storage-initializer"
Sep 18 00:03:50.851545 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.851374    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4629195f-b6ed-47f0-83f5-29c217c5ef16" containerName="storage-initializer"
Sep 18 00:03:50.851545 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.851482    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4629195f-b6ed-47f0-83f5-29c217c5ef16" containerName="storage-initializer"
Sep 18 00:03:50.857074 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.857045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.860087 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.860051    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:03:50.860225 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.860140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 00:03:50.860225 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.860178    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 00:03:50.860363 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.860286    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-mdrk7\""
Sep 18 00:03:50.860646 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.860464    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 00:03:50.860777 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.860734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-nxglk\""
Sep 18 00:03:50.861148 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.861124    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 00:03:50.871030 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.871007    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"]
Sep 18 00:03:50.889701 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.889675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-model-cache\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.889802 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.889705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.889802 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.889736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67febce4-15f4-4db3-9820-1002631c06fa-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.889802 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.889759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.889938 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.889852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-home\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.889938 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.889882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7vm\" (UniqueName: \"kubernetes.io/projected/67febce4-15f4-4db3-9820-1002631c06fa-kube-api-access-fv7vm\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.889938 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.889910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-dshm\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.890054 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.890005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67febce4-15f4-4db3-9820-1002631c06fa-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990450 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67febce4-15f4-4db3-9820-1002631c06fa-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990541 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-model-cache\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990541 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990541 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67febce4-15f4-4db3-9820-1002631c06fa-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990737 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990737 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-home\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990737 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7vm\" (UniqueName: \"kubernetes.io/projected/67febce4-15f4-4db3-9820-1002631c06fa-kube-api-access-fv7vm\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990737 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-dshm\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.990935 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.990915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-model-cache\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.991037 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.991013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.991103 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.991066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-home\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.991183 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.991160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.991462 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.991444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67febce4-15f4-4db3-9820-1002631c06fa-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.992960 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.992942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-dshm\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.993364 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.993341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67febce4-15f4-4db3-9820-1002631c06fa-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:50.998900 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:50.998881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7vm\" (UniqueName: \"kubernetes.io/projected/67febce4-15f4-4db3-9820-1002631c06fa-kube-api-access-fv7vm\") pod \"custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:51.169554 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:51.169484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:51.450381 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:51.450356    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"]
Sep 18 00:03:51.453076 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:03:51.453033    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67febce4_15f4_4db3_9820_1002631c06fa.slice/crio-4a7a8bd9485be8e65a452402f2f48e8fcf5dc32edb07a8d1508f62db158e9889 WatchSource:0}: Error finding container 4a7a8bd9485be8e65a452402f2f48e8fcf5dc32edb07a8d1508f62db158e9889: Status 404 returned error can't find the container with id 4a7a8bd9485be8e65a452402f2f48e8fcf5dc32edb07a8d1508f62db158e9889
Sep 18 00:03:52.309994 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:52.309960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerStarted","Data":"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"}
Sep 18 00:03:52.310388 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:52.310014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerStarted","Data":"4a7a8bd9485be8e65a452402f2f48e8fcf5dc32edb07a8d1508f62db158e9889"}
Sep 18 00:03:52.310388 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:52.310045    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:03:53.322254 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:53.322219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerStarted","Data":"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"}
Sep 18 00:03:55.334083 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:55.334043    2566 generic.go:356] "Generic (PLEG): container finished" podID="67febce4-15f4-4db3-9820-1002631c06fa" containerID="d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f" exitCode=0
Sep 18 00:03:55.334484 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:55.334115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerDied","Data":"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"}
Sep 18 00:03:56.341111 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:56.341076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerStarted","Data":"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"}
Sep 18 00:03:56.365176 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:03:56.365126    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podStartSLOduration=4.201954729 podStartE2EDuration="6.365112116s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.163157387s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:03:56.364708161 +0000 UTC m=+3196.816539648" watchObservedRunningTime="2026-09-18 00:03:56.365112116 +0000 UTC m=+3196.816943604"
Sep 18 00:04:01.170652 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:01.170613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:04:01.171129 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:01.170688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:04:01.171985 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:01.171946    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:04:01.185670 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:01.185646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:04:11.170148 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:11.170106    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:04:21.170242 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:21.170202    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:04:31.170454 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:31.170402    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:04:41.170300 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:41.170207    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:04:51.170428 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:04:51.170369    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:05:01.170567 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:01.170523    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:05:11.170859 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:11.170807    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:05:21.170310 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:21.170262    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:05:31.170734 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:31.170683    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:05:40.287225 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:40.287192    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:05:40.295334 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:40.295310    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:05:40.297956 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:40.297932    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:05:40.305357 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:40.305337    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:05:41.170131 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:41.170081    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 18 00:05:51.182092 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:51.182051    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:05:51.198156 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:51.198123    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:05:54.200854 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:54.200820    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"]
Sep 18 00:05:54.203405 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:54.203378    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-w5lp4"]
Sep 18 00:05:56.013678 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:05:56.013643    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4629195f-b6ed-47f0-83f5-29c217c5ef16" path="/var/lib/kubelet/pods/4629195f-b6ed-47f0-83f5-29c217c5ef16/volumes"
Sep 18 00:06:34.647104 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:34.647069    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"]
Sep 18 00:06:49.670414 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:49.670354    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main" containerID="cri-o://01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4" gracePeriod=45
Sep 18 00:06:50.290294 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.290262    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="llm-d-routing-sidecar" containerID="cri-o://00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982" gracePeriod=45
Sep 18 00:06:50.531328 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.531301    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:06:50.607247 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607219    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-tmp-dir\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607395 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607253    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-kserve-provision-location\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607395 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607280    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67febce4-15f4-4db3-9820-1002631c06fa-cabundle-cert\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607395 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-dshm\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607395 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607369    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67febce4-15f4-4db3-9820-1002631c06fa-tls-certs\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607395 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607391    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-model-cache\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607716 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607413    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-home\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607716 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv7vm\" (UniqueName: \"kubernetes.io/projected/67febce4-15f4-4db3-9820-1002631c06fa-kube-api-access-fv7vm\") pod \"67febce4-15f4-4db3-9820-1002631c06fa\" (UID: \"67febce4-15f4-4db3-9820-1002631c06fa\") "
Sep 18 00:06:50.607834 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67febce4-15f4-4db3-9820-1002631c06fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:06:50.607834 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.607773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.608228 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.608199    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-home" (OuterVolumeSpecName: "home") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.609828 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.609802    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67febce4-15f4-4db3-9820-1002631c06fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:06:50.609950 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.609929    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-dshm" (OuterVolumeSpecName: "dshm") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.610224 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.610206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67febce4-15f4-4db3-9820-1002631c06fa-kube-api-access-fv7vm" (OuterVolumeSpecName: "kube-api-access-fv7vm") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "kube-api-access-fv7vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:06:50.619689 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.619668    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.632240 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.632217    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67febce4-15f4-4db3-9820-1002631c06fa" (UID: "67febce4-15f4-4db3-9820-1002631c06fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708807    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67febce4-15f4-4db3-9820-1002631c06fa-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708835    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708848    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708863    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv7vm\" (UniqueName: \"kubernetes.io/projected/67febce4-15f4-4db3-9820-1002631c06fa-kube-api-access-fv7vm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708884    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708898    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708911    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67febce4-15f4-4db3-9820-1002631c06fa-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.714214 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:50.708923    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67febce4-15f4-4db3-9820-1002631c06fa-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:06:51.207457 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.207413    2566 generic.go:356] "Generic (PLEG): container finished" podID="67febce4-15f4-4db3-9820-1002631c06fa" containerID="01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4" exitCode=0
Sep 18 00:06:51.207457 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.207447    2566 generic.go:356] "Generic (PLEG): container finished" podID="67febce4-15f4-4db3-9820-1002631c06fa" containerID="00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982" exitCode=0
Sep 18 00:06:51.207633 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.207496    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"
Sep 18 00:06:51.207633 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.207487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerDied","Data":"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"}
Sep 18 00:06:51.207722 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.207652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerDied","Data":"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"}
Sep 18 00:06:51.207722 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.207685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4" event={"ID":"67febce4-15f4-4db3-9820-1002631c06fa","Type":"ContainerDied","Data":"4a7a8bd9485be8e65a452402f2f48e8fcf5dc32edb07a8d1508f62db158e9889"}
Sep 18 00:06:51.207722 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.207704    2566 scope.go:117] "RemoveContainer" containerID="01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"
Sep 18 00:06:51.218171 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.218152    2566 scope.go:117] "RemoveContainer" containerID="d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"
Sep 18 00:06:51.226871 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.226850    2566 scope.go:117] "RemoveContainer" containerID="00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"
Sep 18 00:06:51.231971 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.231948    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"]
Sep 18 00:06:51.235509 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.235487    2566 scope.go:117] "RemoveContainer" containerID="01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"
Sep 18 00:06:51.235903 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:06:51.235876    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4\": container with ID starting with 01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4 not found: ID does not exist" containerID="01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"
Sep 18 00:06:51.236009 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.235908    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"} err="failed to get container status \"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4\": rpc error: code = NotFound desc = could not find container \"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4\": container with ID starting with 01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4 not found: ID does not exist"
Sep 18 00:06:51.236009 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.235930    2566 scope.go:117] "RemoveContainer" containerID="d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"
Sep 18 00:06:51.236336 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:06:51.236310    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f\": container with ID starting with d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f not found: ID does not exist" containerID="d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"
Sep 18 00:06:51.236493 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.236345    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"} err="failed to get container status \"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f\": rpc error: code = NotFound desc = could not find container \"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f\": container with ID starting with d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f not found: ID does not exist"
Sep 18 00:06:51.236493 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.236366    2566 scope.go:117] "RemoveContainer" containerID="00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"
Sep 18 00:06:51.237025 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:06:51.236994    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982\": container with ID starting with 00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982 not found: ID does not exist" containerID="00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"
Sep 18 00:06:51.237136 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.237031    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"} err="failed to get container status \"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982\": rpc error: code = NotFound desc = could not find container \"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982\": container with ID starting with 00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982 not found: ID does not exist"
Sep 18 00:06:51.237136 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.237053    2566 scope.go:117] "RemoveContainer" containerID="01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"
Sep 18 00:06:51.237362 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.237336    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4"} err="failed to get container status \"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4\": rpc error: code = NotFound desc = could not find container \"01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4\": container with ID starting with 01c9e1dbf6f47f544d77c0626e3223b9319cba77588633b1dff171ee4730dcf4 not found: ID does not exist"
Sep 18 00:06:51.237414 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.237362    2566 scope.go:117] "RemoveContainer" containerID="d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"
Sep 18 00:06:51.237688 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.237657    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f"} err="failed to get container status \"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f\": rpc error: code = NotFound desc = could not find container \"d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f\": container with ID starting with d9256d6be9e70fb826366a097dc5450e541a161c8a98e0c3b940437320df1e5f not found: ID does not exist"
Sep 18 00:06:51.237688 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.237689    2566 scope.go:117] "RemoveContainer" containerID="00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"
Sep 18 00:06:51.237978 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.237953    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982"} err="failed to get container status \"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982\": rpc error: code = NotFound desc = could not find container \"00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982\": container with ID starting with 00e72443898afdb57560e01acb6becbe52d84307be753dafe4c599a38556e982 not found: ID does not exist"
Sep 18 00:06:51.239178 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:51.239153    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-86c58f596c-8qlp4"]
Sep 18 00:06:52.015258 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:06:52.015223    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67febce4-15f4-4db3-9820-1002631c06fa" path="/var/lib/kubelet/pods/67febce4-15f4-4db3-9820-1002631c06fa/volumes"
Sep 18 00:07:37.862760 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.862719    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"]
Sep 18 00:07:37.863314 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863185    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="llm-d-routing-sidecar"
Sep 18 00:07:37.863314 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863206    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="llm-d-routing-sidecar"
Sep 18 00:07:37.863314 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863234    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="storage-initializer"
Sep 18 00:07:37.863314 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863244    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="storage-initializer"
Sep 18 00:07:37.863314 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863275    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main"
Sep 18 00:07:37.863314 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863285    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main"
Sep 18 00:07:37.863693 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863371    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="llm-d-routing-sidecar"
Sep 18 00:07:37.863693 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.863389    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="67febce4-15f4-4db3-9820-1002631c06fa" containerName="main"
Sep 18 00:07:37.866746 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.866723    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.870409 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.870385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-b9xc2\""
Sep 18 00:07:37.870638 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.870607    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 00:07:37.870638 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.870631    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:07:37.870831 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.870777    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-gxpf2\""
Sep 18 00:07:37.870831 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.870814    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 00:07:37.871515 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.871496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 00:07:37.871606 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.871523    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 00:07:37.895349 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.895318    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"]
Sep 18 00:07:37.953467 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tls-certs\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.953618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9jc\" (UniqueName: \"kubernetes.io/projected/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kube-api-access-bs9jc\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.953618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-dshm\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.953618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-home\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.953618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-model-cache\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.953794 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.953794 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406466cf-0fad-4a9e-b7d5-5d6f473745bf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:37.953794 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:37.953691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054536 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054695 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tls-certs\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054695 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9jc\" (UniqueName: \"kubernetes.io/projected/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kube-api-access-bs9jc\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054695 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-dshm\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054695 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-home\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054695 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-model-cache\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054989 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.054989 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406466cf-0fad-4a9e-b7d5-5d6f473745bf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.055105 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.054982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.055250 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.055222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-home\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.055396 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.055373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.055493 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.055472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-model-cache\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.055864 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.055839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406466cf-0fad-4a9e-b7d5-5d6f473745bf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.056863 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.056844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-dshm\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.057271 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.057251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tls-certs\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.062965 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.062938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9jc\" (UniqueName: \"kubernetes.io/projected/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kube-api-access-bs9jc\") pod \"router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.176661 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.176601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:38.453190 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:38.453158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"]
Sep 18 00:07:38.454884 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:07:38.454859    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406466cf_0fad_4a9e_b7d5_5d6f473745bf.slice/crio-2bd8054281daced3498687bc350600a16fc97e8abe6ace9ac4efcc779efd1f4b WatchSource:0}: Error finding container 2bd8054281daced3498687bc350600a16fc97e8abe6ace9ac4efcc779efd1f4b: Status 404 returned error can't find the container with id 2bd8054281daced3498687bc350600a16fc97e8abe6ace9ac4efcc779efd1f4b
Sep 18 00:07:39.451783 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:39.451748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerStarted","Data":"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"}
Sep 18 00:07:39.452268 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:39.451800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerStarted","Data":"2bd8054281daced3498687bc350600a16fc97e8abe6ace9ac4efcc779efd1f4b"}
Sep 18 00:07:39.452268 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:39.451891    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:40.461313 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:40.461254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerStarted","Data":"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"}
Sep 18 00:07:42.474476 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:42.474431    2566 generic.go:356] "Generic (PLEG): container finished" podID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerID="db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04" exitCode=0
Sep 18 00:07:42.474961 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:42.474500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerDied","Data":"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"}
Sep 18 00:07:43.481132 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:43.481055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerStarted","Data":"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"}
Sep 18 00:07:43.507803 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:43.507746    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podStartSLOduration=4.346743223 podStartE2EDuration="6.507725259s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.160982036s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:07:43.50503581 +0000 UTC m=+3423.956867299" watchObservedRunningTime="2026-09-18 00:07:43.507725259 +0000 UTC m=+3423.959556748"
Sep 18 00:07:48.177682 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:48.177645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:48.178135 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:48.177715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:48.178953 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:48.178921    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:07:48.191304 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:48.191278    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:07:58.177324 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:07:58.177276    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:08:08.177919 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:08.177872    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:08:18.177408 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:18.177358    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:08:28.177423 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:28.177366    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:08:38.177538 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:38.177492    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:08:48.177971 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:48.177912    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:08:49.457632 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.457591    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"]
Sep 18 00:08:49.461677 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.461656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:49.464368 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.464341    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-rspsw\""
Sep 18 00:08:49.465544 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.465517    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 18 00:08:49.465915 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.465860    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 18 00:08:49.466019 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.465860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 00:08:49.466098 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.466078    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-s5dlv\""
Sep 18 00:08:49.468441 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.468418    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"]
Sep 18 00:08:49.552223 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.552197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23626b42-d226-4166-b6c3-7c957d209118-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:49.552347 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.552230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8kr\" (UniqueName: \"kubernetes.io/projected/23626b42-d226-4166-b6c3-7c957d209118-kube-api-access-cg8kr\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:49.653753 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.653723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23626b42-d226-4166-b6c3-7c957d209118-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:49.653753 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.653758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8kr\" (UniqueName: \"kubernetes.io/projected/23626b42-d226-4166-b6c3-7c957d209118-kube-api-access-cg8kr\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:49.656561 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.656535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23626b42-d226-4166-b6c3-7c957d209118-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:49.661615 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.661569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8kr\" (UniqueName: \"kubernetes.io/projected/23626b42-d226-4166-b6c3-7c957d209118-kube-api-access-cg8kr\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:49.775248 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:49.775224    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:50.051407 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:50.051315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"]
Sep 18 00:08:50.055198 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:08:50.055140    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23626b42_d226_4166_b6c3_7c957d209118.slice/crio-41f36f850e434d5e7043c8ce8575d6e2be4f58fa7cde9a2ceb913c97b9adccc9 WatchSource:0}: Error finding container 41f36f850e434d5e7043c8ce8575d6e2be4f58fa7cde9a2ceb913c97b9adccc9: Status 404 returned error can't find the container with id 41f36f850e434d5e7043c8ce8575d6e2be4f58fa7cde9a2ceb913c97b9adccc9
Sep 18 00:08:50.831047 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:50.831010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" event={"ID":"23626b42-d226-4166-b6c3-7c957d209118","Type":"ContainerStarted","Data":"bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc"}
Sep 18 00:08:50.831601 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:50.831067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" event={"ID":"23626b42-d226-4166-b6c3-7c957d209118","Type":"ContainerStarted","Data":"41f36f850e434d5e7043c8ce8575d6e2be4f58fa7cde9a2ceb913c97b9adccc9"}
Sep 18 00:08:50.831601 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:50.831093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:08:50.848436 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:50.848387    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" podStartSLOduration=1.848372546 podStartE2EDuration="1.848372546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:08:50.846027384 +0000 UTC m=+3491.297858883" watchObservedRunningTime="2026-09-18 00:08:50.848372546 +0000 UTC m=+3491.300204034"
Sep 18 00:08:58.177927 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:08:58.177880    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:09:08.177180 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:08.177073    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:09:18.177077 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:18.177037    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:09:21.844444 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:21.844410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:09:28.177383 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:28.177323    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 18 00:09:38.188306 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:38.188265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:09:38.206172 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:38.206148    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:09:46.166839 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:46.166807    2566 scope.go:117] "RemoveContainer" containerID="00526560d7ff95949edb52638ead9b064bc314944754eab568efd845ed0b8226"
Sep 18 00:09:47.054795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:09:47.054755    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"]
Sep 18 00:10:02.085836 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:02.085794    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main" containerID="cri-o://75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07" gracePeriod=45
Sep 18 00:10:02.701508 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:02.701458    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="llm-d-routing-sidecar" containerID="cri-o://1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89" gracePeriod=45
Sep 18 00:10:02.942728 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:02.942705    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:10:03.052812 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.052786    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs9jc\" (UniqueName: \"kubernetes.io/projected/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kube-api-access-bs9jc\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.052970 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.052819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406466cf-0fad-4a9e-b7d5-5d6f473745bf-cabundle-cert\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.052970 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.052855    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tmp-dir\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.053097 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.052976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kserve-provision-location\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.053097 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-dshm\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.053097 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tls-certs\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.053252 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053131    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-home\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.053252 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-model-cache\") pod \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\" (UID: \"406466cf-0fad-4a9e-b7d5-5d6f473745bf\") "
Sep 18 00:10:03.053354 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053269    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406466cf-0fad-4a9e-b7d5-5d6f473745bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:10:03.053717 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053433    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406466cf-0fad-4a9e-b7d5-5d6f473745bf-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.053717 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053706    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:03.053982 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.053954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-home" (OuterVolumeSpecName: "home") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:03.055289 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.055265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kube-api-access-bs9jc" (OuterVolumeSpecName: "kube-api-access-bs9jc") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "kube-api-access-bs9jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:10:03.055741 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.055711    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:10:03.055825 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.055788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-dshm" (OuterVolumeSpecName: "dshm") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:03.065523 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.065501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:03.078044 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.078021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "406466cf-0fad-4a9e-b7d5-5d6f473745bf" (UID: "406466cf-0fad-4a9e-b7d5-5d6f473745bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:03.154353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.154294    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9jc\" (UniqueName: \"kubernetes.io/projected/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kube-api-access-bs9jc\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.154353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.154318    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.154353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.154329    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.154353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.154338    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.154353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.154347    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/406466cf-0fad-4a9e-b7d5-5d6f473745bf-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.154353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.154355    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.154353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.154365    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/406466cf-0fad-4a9e-b7d5-5d6f473745bf-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.186854 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.186822    2566 generic.go:356] "Generic (PLEG): container finished" podID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerID="75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07" exitCode=0
Sep 18 00:10:03.186854 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.186849    2566 generic.go:356] "Generic (PLEG): container finished" podID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerID="1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89" exitCode=0
Sep 18 00:10:03.186996 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.186876    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"
Sep 18 00:10:03.186996 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.186870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerDied","Data":"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"}
Sep 18 00:10:03.186996 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.186991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerDied","Data":"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"}
Sep 18 00:10:03.187110 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.187011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8" event={"ID":"406466cf-0fad-4a9e-b7d5-5d6f473745bf","Type":"ContainerDied","Data":"2bd8054281daced3498687bc350600a16fc97e8abe6ace9ac4efcc779efd1f4b"}
Sep 18 00:10:03.187110 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.187032    2566 scope.go:117] "RemoveContainer" containerID="75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"
Sep 18 00:10:03.196140 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.196124    2566 scope.go:117] "RemoveContainer" containerID="db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"
Sep 18 00:10:03.203594 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.203552    2566 scope.go:117] "RemoveContainer" containerID="1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"
Sep 18 00:10:03.209767 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.209744    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"]
Sep 18 00:10:03.211107 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.211075    2566 scope.go:117] "RemoveContainer" containerID="75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"
Sep 18 00:10:03.211520 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:10:03.211492    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07\": container with ID starting with 75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07 not found: ID does not exist" containerID="75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"
Sep 18 00:10:03.211618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.211532    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"} err="failed to get container status \"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07\": rpc error: code = NotFound desc = could not find container \"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07\": container with ID starting with 75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07 not found: ID does not exist"
Sep 18 00:10:03.211618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.211557    2566 scope.go:117] "RemoveContainer" containerID="db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"
Sep 18 00:10:03.212084 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:10:03.212063    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04\": container with ID starting with db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04 not found: ID does not exist" containerID="db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"
Sep 18 00:10:03.212179 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212089    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"} err="failed to get container status \"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04\": rpc error: code = NotFound desc = could not find container \"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04\": container with ID starting with db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04 not found: ID does not exist"
Sep 18 00:10:03.212179 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212104    2566 scope.go:117] "RemoveContainer" containerID="1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"
Sep 18 00:10:03.212386 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:10:03.212367    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89\": container with ID starting with 1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89 not found: ID does not exist" containerID="1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"
Sep 18 00:10:03.212444 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212391    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"} err="failed to get container status \"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89\": rpc error: code = NotFound desc = could not find container \"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89\": container with ID starting with 1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89 not found: ID does not exist"
Sep 18 00:10:03.212444 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212407    2566 scope.go:117] "RemoveContainer" containerID="75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"
Sep 18 00:10:03.212689 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212661    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07"} err="failed to get container status \"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07\": rpc error: code = NotFound desc = could not find container \"75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07\": container with ID starting with 75ee47a333c75e53dcde05ddb30f64179739d868aae3cb9f4a97cf1e0febbf07 not found: ID does not exist"
Sep 18 00:10:03.212755 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212691    2566 scope.go:117] "RemoveContainer" containerID="db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"
Sep 18 00:10:03.212974 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212955    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04"} err="failed to get container status \"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04\": rpc error: code = NotFound desc = could not find container \"db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04\": container with ID starting with db2c76b695188a0dc9fd04decdd3f983004e8e2e1032a3e2c2fbe2b4becd7a04 not found: ID does not exist"
Sep 18 00:10:03.213036 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.212974    2566 scope.go:117] "RemoveContainer" containerID="1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"
Sep 18 00:10:03.213230 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.213202    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89"} err="failed to get container status \"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89\": rpc error: code = NotFound desc = could not find container \"1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89\": container with ID starting with 1d07699c79f63a3ba3cf759bc2b8f345c997e5d67ca7a2a41ad752c1e8ae6d89 not found: ID does not exist"
Sep 18 00:10:03.213428 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:03.213411    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6bd4f89db5-j8bw8"]
Sep 18 00:10:04.013874 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:04.013839    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" path="/var/lib/kubelet/pods/406466cf-0fad-4a9e-b7d5-5d6f473745bf/volumes"
Sep 18 00:10:40.318242 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:40.318202    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:10:40.327339 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:40.327315    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:10:40.329665 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:40.329645    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:10:40.337698 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:40.337678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:10:50.900884 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.900851    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"]
Sep 18 00:10:50.901387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901240    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="llm-d-routing-sidecar"
Sep 18 00:10:50.901387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901256    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="llm-d-routing-sidecar"
Sep 18 00:10:50.901387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901312    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="storage-initializer"
Sep 18 00:10:50.901387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901319    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="storage-initializer"
Sep 18 00:10:50.901387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901327    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main"
Sep 18 00:10:50.901387 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901345    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main"
Sep 18 00:10:50.901673 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901429    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="main"
Sep 18 00:10:50.901673 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.901449    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="406466cf-0fad-4a9e-b7d5-5d6f473745bf" containerName="llm-d-routing-sidecar"
Sep 18 00:10:50.906036 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.906017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.909549 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.909514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 18 00:10:50.909682 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.909554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-2xn4l\""
Sep 18 00:10:50.909682 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.909569    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 18 00:10:50.909682 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.909519    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 18 00:10:50.909682 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.909612    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 18 00:10:50.909899 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.909688    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:10:50.916239 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.916212    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"]
Sep 18 00:10:50.917240 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.917358 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.917358 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.917358 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.917510 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69pt\" (UniqueName: \"kubernetes.io/projected/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kube-api-access-k69pt\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.917510 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.917510 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:50.917648 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:50.917509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f5a00ac-bb66-4954-a434-61359bbdbb6b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.018647 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.018622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.018770 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.018656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.018843 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.018780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69pt\" (UniqueName: \"kubernetes.io/projected/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kube-api-access-k69pt\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.018843 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.018826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019012 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.018990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019104 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019104 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019239 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f5a00ac-bb66-4954-a434-61359bbdbb6b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019300 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019300 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019446 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019687 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019658    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.019758 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.019738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f5a00ac-bb66-4954-a434-61359bbdbb6b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.021276 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.021252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.021559 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.021542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.026526 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.026503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69pt\" (UniqueName: \"kubernetes.io/projected/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kube-api-access-k69pt\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.217877 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.217821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:10:51.508009 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:51.507978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"]
Sep 18 00:10:51.509054 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:10:51.509023    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5a00ac_bb66_4954_a434_61359bbdbb6b.slice/crio-a12ed34d9d0216cc4edda6b66b52e80207d98f8b6a51d249b44840aaf8636a32 WatchSource:0}: Error finding container a12ed34d9d0216cc4edda6b66b52e80207d98f8b6a51d249b44840aaf8636a32: Status 404 returned error can't find the container with id a12ed34d9d0216cc4edda6b66b52e80207d98f8b6a51d249b44840aaf8636a32
Sep 18 00:10:52.434380 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:52.434339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" event={"ID":"6f5a00ac-bb66-4954-a434-61359bbdbb6b","Type":"ContainerStarted","Data":"1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c"}
Sep 18 00:10:52.434805 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:52.434401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" event={"ID":"6f5a00ac-bb66-4954-a434-61359bbdbb6b","Type":"ContainerStarted","Data":"a12ed34d9d0216cc4edda6b66b52e80207d98f8b6a51d249b44840aaf8636a32"}
Sep 18 00:10:54.446409 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:54.446372    2566 generic.go:356] "Generic (PLEG): container finished" podID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerID="1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c" exitCode=0
Sep 18 00:10:54.446810 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:54.446443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" event={"ID":"6f5a00ac-bb66-4954-a434-61359bbdbb6b","Type":"ContainerDied","Data":"1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c"}
Sep 18 00:10:55.453640 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:55.453602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" event={"ID":"6f5a00ac-bb66-4954-a434-61359bbdbb6b","Type":"ContainerStarted","Data":"f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea"}
Sep 18 00:10:55.476992 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:55.476940    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podStartSLOduration=3.160249676 podStartE2EDuration="5.476926175s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.316676499s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:10:55.473839686 +0000 UTC m=+3615.925671174" watchObservedRunningTime="2026-09-18 00:10:55.476926175 +0000 UTC m=+3615.928757662"
Sep 18 00:10:56.218710 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:10:56.218677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:11:01.217995 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:01.217956    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:11:01.219362 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:01.219335    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:11:11.218971 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:11.218924    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:11:21.218196 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:21.218154    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:11:31.218521 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:31.218476    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:11:41.143123 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:41.143086    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"]
Sep 18 00:11:41.218392 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:41.218347    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:11:41.838071 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:41.838023    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:11:51.218859 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:51.218810    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:11:51.838241 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:51.838206    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:11:56.163707 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:11:56.163664    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main" containerID="cri-o://bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc" gracePeriod=45
Sep 18 00:12:01.218924 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:01.218883    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:12:01.837843 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:01.837803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:12:01.838003 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:01.837911    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:12:11.218228 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:11.218127    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:12:11.838277 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:11.838239    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:12:21.218647 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:21.218597    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:12:21.838082 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:21.838041    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:12:26.412202 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.412173    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:12:26.511247 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.511218    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23626b42-d226-4166-b6c3-7c957d209118-tls-certs\") pod \"23626b42-d226-4166-b6c3-7c957d209118\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") "
Sep 18 00:12:26.511371 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.511272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg8kr\" (UniqueName: \"kubernetes.io/projected/23626b42-d226-4166-b6c3-7c957d209118-kube-api-access-cg8kr\") pod \"23626b42-d226-4166-b6c3-7c957d209118\" (UID: \"23626b42-d226-4166-b6c3-7c957d209118\") "
Sep 18 00:12:26.513494 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.513461    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23626b42-d226-4166-b6c3-7c957d209118-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23626b42-d226-4166-b6c3-7c957d209118" (UID: "23626b42-d226-4166-b6c3-7c957d209118"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:12:26.513494 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.513484    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23626b42-d226-4166-b6c3-7c957d209118-kube-api-access-cg8kr" (OuterVolumeSpecName: "kube-api-access-cg8kr") pod "23626b42-d226-4166-b6c3-7c957d209118" (UID: "23626b42-d226-4166-b6c3-7c957d209118"). InnerVolumeSpecName "kube-api-access-cg8kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:12:26.611992 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.611965    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23626b42-d226-4166-b6c3-7c957d209118-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:12:26.611992 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.611991    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg8kr\" (UniqueName: \"kubernetes.io/projected/23626b42-d226-4166-b6c3-7c957d209118-kube-api-access-cg8kr\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:12:26.891304 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.891213    2566 generic.go:356] "Generic (PLEG): container finished" podID="23626b42-d226-4166-b6c3-7c957d209118" containerID="bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc" exitCode=0
Sep 18 00:12:26.891304 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.891264    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"
Sep 18 00:12:26.891545 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.891299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" event={"ID":"23626b42-d226-4166-b6c3-7c957d209118","Type":"ContainerDied","Data":"bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc"}
Sep 18 00:12:26.891545 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.891348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf" event={"ID":"23626b42-d226-4166-b6c3-7c957d209118","Type":"ContainerDied","Data":"41f36f850e434d5e7043c8ce8575d6e2be4f58fa7cde9a2ceb913c97b9adccc9"}
Sep 18 00:12:26.891545 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.891371    2566 scope.go:117] "RemoveContainer" containerID="bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc"
Sep 18 00:12:26.901328 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.901309    2566 scope.go:117] "RemoveContainer" containerID="bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc"
Sep 18 00:12:26.901661 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:12:26.901640    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc\": container with ID starting with bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc not found: ID does not exist" containerID="bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc"
Sep 18 00:12:26.901720 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.901670    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc"} err="failed to get container status \"bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc\": rpc error: code = NotFound desc = could not find container \"bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc\": container with ID starting with bcb33fbbb929686a7c9a1dbfd76896febdf3722f56cf80fc8bcd7d6250c807dc not found: ID does not exist"
Sep 18 00:12:26.912038 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.912006    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"]
Sep 18 00:12:26.915055 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:26.915033    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mjgbf"]
Sep 18 00:12:28.014276 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:28.014244    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23626b42-d226-4166-b6c3-7c957d209118" path="/var/lib/kubelet/pods/23626b42-d226-4166-b6c3-7c957d209118/volumes"
Sep 18 00:12:31.218439 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:31.218399    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:12:41.218770 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:41.218717    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 00:12:51.228840 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:51.228794    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:12:51.237191 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:51.237158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:12:57.426906 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:12:57.426863    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"]
Sep 18 00:13:12.450405 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:12.450344    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main" containerID="cri-o://f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea" gracePeriod=45
Sep 18 00:13:13.096531 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.096505    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:13:13.123185 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.123150    2566 generic.go:356] "Generic (PLEG): container finished" podID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerID="f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea" exitCode=0
Sep 18 00:13:13.123331 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.123198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" event={"ID":"6f5a00ac-bb66-4954-a434-61359bbdbb6b","Type":"ContainerDied","Data":"f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea"}
Sep 18 00:13:13.123331 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.123209    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"
Sep 18 00:13:13.123331 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.123274    2566 scope.go:117] "RemoveContainer" containerID="f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea"
Sep 18 00:13:13.123499 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.123259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm" event={"ID":"6f5a00ac-bb66-4954-a434-61359bbdbb6b","Type":"ContainerDied","Data":"a12ed34d9d0216cc4edda6b66b52e80207d98f8b6a51d249b44840aaf8636a32"}
Sep 18 00:13:13.131635 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.131611    2566 scope.go:117] "RemoveContainer" containerID="1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c"
Sep 18 00:13:13.139030 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.139012    2566 scope.go:117] "RemoveContainer" containerID="f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea"
Sep 18 00:13:13.139320 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:13:13.139298    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea\": container with ID starting with f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea not found: ID does not exist" containerID="f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea"
Sep 18 00:13:13.139399 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.139327    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea"} err="failed to get container status \"f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea\": rpc error: code = NotFound desc = could not find container \"f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea\": container with ID starting with f015ccfcbffb274572019df7eb96a4533fa1a7cc6939752aad05776a619e78ea not found: ID does not exist"
Sep 18 00:13:13.139399 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.139344    2566 scope.go:117] "RemoveContainer" containerID="1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c"
Sep 18 00:13:13.139619 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:13:13.139597    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c\": container with ID starting with 1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c not found: ID does not exist" containerID="1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c"
Sep 18 00:13:13.139710 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.139631    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c"} err="failed to get container status \"1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c\": rpc error: code = NotFound desc = could not find container \"1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c\": container with ID starting with 1474937910ece68a9871b187ed44f62efe05d096f373d7f600c4b1e12b8db71c not found: ID does not exist"
Sep 18 00:13:13.185806 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.185786    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-model-cache\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.185887 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.185837    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69pt\" (UniqueName: \"kubernetes.io/projected/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kube-api-access-k69pt\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.185887 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.185866    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kserve-provision-location\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.185987 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.185902    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tmp-dir\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.185987 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.185941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tls-certs\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.185987 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.185967    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-home\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.186145 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.186001    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f5a00ac-bb66-4954-a434-61359bbdbb6b-cabundle-cert\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.186145 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.186029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-dshm\") pod \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\" (UID: \"6f5a00ac-bb66-4954-a434-61359bbdbb6b\") "
Sep 18 00:13:13.186145 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.186054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:13:13.186307 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.186224    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.186509 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.186480    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5a00ac-bb66-4954-a434-61359bbdbb6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:13:13.186809 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.186783    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-home" (OuterVolumeSpecName: "home") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:13:13.188193 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.188163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:13:13.188352 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.188332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kube-api-access-k69pt" (OuterVolumeSpecName: "kube-api-access-k69pt") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "kube-api-access-k69pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:13:13.188470 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.188454    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-dshm" (OuterVolumeSpecName: "dshm") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:13:13.199145 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.199110    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:13:13.212774 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.212751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f5a00ac-bb66-4954-a434-61359bbdbb6b" (UID: "6f5a00ac-bb66-4954-a434-61359bbdbb6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:13:13.287140 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.287115    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69pt\" (UniqueName: \"kubernetes.io/projected/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kube-api-access-k69pt\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.287140 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.287139    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.287301 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.287154    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.287301 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.287167    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a00ac-bb66-4954-a434-61359bbdbb6b-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.287301 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.287179    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.287301 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.287191    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f5a00ac-bb66-4954-a434-61359bbdbb6b-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.287301 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.287203    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a00ac-bb66-4954-a434-61359bbdbb6b-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:13:13.448998 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.448933    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"]
Sep 18 00:13:13.454497 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:13.454472    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-598f69f9dbh9fnm"]
Sep 18 00:13:14.014402 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:14.014366    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" path="/var/lib/kubelet/pods/6f5a00ac-bb66-4954-a434-61359bbdbb6b/volumes"
Sep 18 00:13:27.277565 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277527    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 00:13:27.277941 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277854    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main"
Sep 18 00:13:27.277941 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277867    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main"
Sep 18 00:13:27.277941 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277886    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="storage-initializer"
Sep 18 00:13:27.277941 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277892    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="storage-initializer"
Sep 18 00:13:27.277941 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277903    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main"
Sep 18 00:13:27.277941 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277910    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main"
Sep 18 00:13:27.278245 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277959    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="23626b42-d226-4166-b6c3-7c957d209118" containerName="main"
Sep 18 00:13:27.278245 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.277967    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f5a00ac-bb66-4954-a434-61359bbdbb6b" containerName="main"
Sep 18 00:13:27.281379 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.281358    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.284076 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.284052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 00:13:27.284178 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.284139    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 00:13:27.285284 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.285266    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 00:13:27.285365 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.285310    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-4lzz4\""
Sep 18 00:13:27.285365 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.285349    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:13:27.285480 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.285365    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 00:13:27.285480 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.285429    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-4cwsc\""
Sep 18 00:13:27.292918 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.292897    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 00:13:27.366750 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.366719    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"]
Sep 18 00:13:27.369388 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.369369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.371967 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.371947    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-cwg8r\""
Sep 18 00:13:27.379028 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.379004    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"]
Sep 18 00:13:27.398056 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.398148 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.398148 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.398227 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.398227 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1333d310-cca3-417a-8fea-ef8a51ebc428-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.398296 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.398296 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1333d310-cca3-417a-8fea-ef8a51ebc428-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.398369 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.398296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbh9\" (UniqueName: \"kubernetes.io/projected/1333d310-cca3-417a-8fea-ef8a51ebc428-kube-api-access-9nbh9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499383 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499482 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499482 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1333d310-cca3-417a-8fea-ef8a51ebc428-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499482 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499628 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jqs\" (UniqueName: \"kubernetes.io/projected/ae3ba79e-77fd-4783-ad46-1b92b15b8111-kube-api-access-q2jqs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.499628 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1333d310-cca3-417a-8fea-ef8a51ebc428-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499717 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbh9\" (UniqueName: \"kubernetes.io/projected/1333d310-cca3-417a-8fea-ef8a51ebc428-kube-api-access-9nbh9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499770 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3ba79e-77fd-4783-ad46-1b92b15b8111-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.499770 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499877 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499877 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.499988 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.499970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.500046 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.500031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.500099 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.500085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.500194 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.500173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1333d310-cca3-417a-8fea-ef8a51ebc428-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.501680 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.501662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.501855 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.501835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1333d310-cca3-417a-8fea-ef8a51ebc428-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.507482 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.507459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbh9\" (UniqueName: \"kubernetes.io/projected/1333d310-cca3-417a-8fea-ef8a51ebc428-kube-api-access-9nbh9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.591462 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.591398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:27.601044 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.601023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jqs\" (UniqueName: \"kubernetes.io/projected/ae3ba79e-77fd-4783-ad46-1b92b15b8111-kube-api-access-q2jqs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.601116 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.601066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3ba79e-77fd-4783-ad46-1b92b15b8111-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.603404 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.603384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3ba79e-77fd-4783-ad46-1b92b15b8111-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.608642 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.608619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jqs\" (UniqueName: \"kubernetes.io/projected/ae3ba79e-77fd-4783-ad46-1b92b15b8111-kube-api-access-q2jqs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.680542 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.680517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:27.870131 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.870105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 00:13:27.872119 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:13:27.872091    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1333d310_cca3_417a_8fea_ef8a51ebc428.slice/crio-02ab92771e9d790b93814a78a961938b4b8dfb256f4048cde008a64b4b954edc WatchSource:0}: Error finding container 02ab92771e9d790b93814a78a961938b4b8dfb256f4048cde008a64b4b954edc: Status 404 returned error can't find the container with id 02ab92771e9d790b93814a78a961938b4b8dfb256f4048cde008a64b4b954edc
Sep 18 00:13:27.954350 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:27.954323    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"]
Sep 18 00:13:27.957806 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:13:27.957777    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3ba79e_77fd_4783_ad46_1b92b15b8111.slice/crio-7d60bb53ae4e2abf3df18bc0c4bb214c51ee8c9b1396d4b916f44af6209ffc48 WatchSource:0}: Error finding container 7d60bb53ae4e2abf3df18bc0c4bb214c51ee8c9b1396d4b916f44af6209ffc48: Status 404 returned error can't find the container with id 7d60bb53ae4e2abf3df18bc0c4bb214c51ee8c9b1396d4b916f44af6209ffc48
Sep 18 00:13:28.199612 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:28.199493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" event={"ID":"ae3ba79e-77fd-4783-ad46-1b92b15b8111","Type":"ContainerStarted","Data":"c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5"}
Sep 18 00:13:28.199612 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:28.199555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" event={"ID":"ae3ba79e-77fd-4783-ad46-1b92b15b8111","Type":"ContainerStarted","Data":"7d60bb53ae4e2abf3df18bc0c4bb214c51ee8c9b1396d4b916f44af6209ffc48"}
Sep 18 00:13:28.199612 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:28.199599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:13:28.201409 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:28.201376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"1333d310-cca3-417a-8fea-ef8a51ebc428","Type":"ContainerStarted","Data":"320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a"}
Sep 18 00:13:28.201553 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:28.201425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"1333d310-cca3-417a-8fea-ef8a51ebc428","Type":"ContainerStarted","Data":"02ab92771e9d790b93814a78a961938b4b8dfb256f4048cde008a64b4b954edc"}
Sep 18 00:13:28.216909 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:28.216856    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" podStartSLOduration=1.216838228 podStartE2EDuration="1.216838228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:13:28.216517238 +0000 UTC m=+3768.668348726" watchObservedRunningTime="2026-09-18 00:13:28.216838228 +0000 UTC m=+3768.668669720"
Sep 18 00:13:31.216569 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:31.216533    2566 generic.go:356] "Generic (PLEG): container finished" podID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerID="320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a" exitCode=0
Sep 18 00:13:31.217050 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:31.216611    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"1333d310-cca3-417a-8fea-ef8a51ebc428","Type":"ContainerDied","Data":"320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a"}
Sep 18 00:13:32.224827 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:32.224789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"1333d310-cca3-417a-8fea-ef8a51ebc428","Type":"ContainerStarted","Data":"1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2"}
Sep 18 00:13:32.247704 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:32.247654    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.879401427 podStartE2EDuration="5.247638898s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.368237471s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:13:32.244654413 +0000 UTC m=+3772.696485901" watchObservedRunningTime="2026-09-18 00:13:32.247638898 +0000 UTC m=+3772.699470389"
Sep 18 00:13:32.591738 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:32.591705    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:37.592223 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:37.592182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:13:37.593671 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:37.593634    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:13:47.592386 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:47.592342    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:13:57.592694 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:57.592644    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:13:59.208163 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:13:59.208128    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:14:07.592762 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:14:07.592718    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:14:17.592045 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:14:17.592003    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:14:27.592100 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:14:27.592050    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:14:37.592171 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:14:37.592122    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:14:47.592674 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:14:47.592622    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:14:57.592178 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:14:57.592135    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:15:07.601511 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:07.601434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:15:07.608904 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:07.608881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:15:16.454215 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:16.454180    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"]
Sep 18 00:15:16.462601 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:16.462553    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 00:15:19.208188 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:19.208150    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:15:29.208272 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:29.208220    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:15:31.478513 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:31.478461    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main" containerID="cri-o://c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5" gracePeriod=45
Sep 18 00:15:31.493394 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:31.493348    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" containerID="cri-o://1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2" gracePeriod=45
Sep 18 00:15:31.602712 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:31.602672    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 00:15:32.153731 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.153703    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:15:32.223319 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223287    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-kserve-provision-location\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.223451 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-dshm\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.223451 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223402    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nbh9\" (UniqueName: \"kubernetes.io/projected/1333d310-cca3-417a-8fea-ef8a51ebc428-kube-api-access-9nbh9\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.223451 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-tmp-dir\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.223672 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223488    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-home\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.223672 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223520    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1333d310-cca3-417a-8fea-ef8a51ebc428-tls-certs\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.223672 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223543    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1333d310-cca3-417a-8fea-ef8a51ebc428-cabundle-cert\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.223672 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.223618    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-model-cache\") pod \"1333d310-cca3-417a-8fea-ef8a51ebc428\" (UID: \"1333d310-cca3-417a-8fea-ef8a51ebc428\") "
Sep 18 00:15:32.224267 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.224238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-model-cache" (OuterVolumeSpecName: "model-cache") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:32.224370 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.224267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1333d310-cca3-417a-8fea-ef8a51ebc428-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:15:32.224370 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.224295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-home" (OuterVolumeSpecName: "home") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:32.225774 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.225733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-dshm" (OuterVolumeSpecName: "dshm") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:32.226291 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.226270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1333d310-cca3-417a-8fea-ef8a51ebc428-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:15:32.226618 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.226601    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1333d310-cca3-417a-8fea-ef8a51ebc428-kube-api-access-9nbh9" (OuterVolumeSpecName: "kube-api-access-9nbh9") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "kube-api-access-9nbh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:15:32.236882 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.236826    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:32.249120 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.249094    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1333d310-cca3-417a-8fea-ef8a51ebc428" (UID: "1333d310-cca3-417a-8fea-ef8a51ebc428"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:32.324451 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324416    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-kserve-provision-location\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.324451 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324449    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.324451 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324459    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nbh9\" (UniqueName: \"kubernetes.io/projected/1333d310-cca3-417a-8fea-ef8a51ebc428-kube-api-access-9nbh9\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.324638 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324469    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.324638 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324483    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.324638 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324493    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1333d310-cca3-417a-8fea-ef8a51ebc428-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.324638 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324501    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1333d310-cca3-417a-8fea-ef8a51ebc428-cabundle-cert\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.324638 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.324508    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1333d310-cca3-417a-8fea-ef8a51ebc428-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:15:32.822396 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.822362    2566 generic.go:356] "Generic (PLEG): container finished" podID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerID="1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2" exitCode=0
Sep 18 00:15:32.822789 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.822417    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 00:15:32.822789 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.822453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"1333d310-cca3-417a-8fea-ef8a51ebc428","Type":"ContainerDied","Data":"1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2"}
Sep 18 00:15:32.822789 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.822526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"1333d310-cca3-417a-8fea-ef8a51ebc428","Type":"ContainerDied","Data":"02ab92771e9d790b93814a78a961938b4b8dfb256f4048cde008a64b4b954edc"}
Sep 18 00:15:32.822789 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.822558    2566 scope.go:117] "RemoveContainer" containerID="1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2"
Sep 18 00:15:32.832603 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.832552    2566 scope.go:117] "RemoveContainer" containerID="320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a"
Sep 18 00:15:32.840917 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.840896    2566 scope.go:117] "RemoveContainer" containerID="1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2"
Sep 18 00:15:32.841225 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:15:32.841198    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2\": container with ID starting with 1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2 not found: ID does not exist" containerID="1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2"
Sep 18 00:15:32.841281 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.841234    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2"} err="failed to get container status \"1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2\": rpc error: code = NotFound desc = could not find container \"1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2\": container with ID starting with 1a6699cf33d52f788212482208977be57c99fd46574bde8ca74e9b92c6bc51e2 not found: ID does not exist"
Sep 18 00:15:32.841281 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.841254    2566 scope.go:117] "RemoveContainer" containerID="320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a"
Sep 18 00:15:32.841535 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:15:32.841514    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a\": container with ID starting with 320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a not found: ID does not exist" containerID="320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a"
Sep 18 00:15:32.841623 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.841545    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a"} err="failed to get container status \"320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a\": rpc error: code = NotFound desc = could not find container \"320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a\": container with ID starting with 320238b527ec9b23cd75d8b55cf060079ffd83b6356a206d98514276003cab4a not found: ID does not exist"
Sep 18 00:15:32.846046 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.846020    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 00:15:32.851959 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:32.851934    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 00:15:34.014692 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:34.014652    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" path="/var/lib/kubelet/pods/1333d310-cca3-417a-8fea-ef8a51ebc428/volumes"
Sep 18 00:15:39.207825 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:39.207787    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:15:39.208245 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:39.207915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:15:40.349769 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:40.349734    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:15:40.357471 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:40.357445    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:15:40.360888 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:40.360867    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:15:40.368537 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:40.368519    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:15:49.207976 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:49.207929    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:15:59.207688 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:15:59.207644    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:16:01.730985 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.730961    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:16:01.752868 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.752808    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2jqs\" (UniqueName: \"kubernetes.io/projected/ae3ba79e-77fd-4783-ad46-1b92b15b8111-kube-api-access-q2jqs\") pod \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") "
Sep 18 00:16:01.752868 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.752849    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3ba79e-77fd-4783-ad46-1b92b15b8111-tls-certs\") pod \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\" (UID: \"ae3ba79e-77fd-4783-ad46-1b92b15b8111\") "
Sep 18 00:16:01.754881 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.754850    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3ba79e-77fd-4783-ad46-1b92b15b8111-kube-api-access-q2jqs" (OuterVolumeSpecName: "kube-api-access-q2jqs") pod "ae3ba79e-77fd-4783-ad46-1b92b15b8111" (UID: "ae3ba79e-77fd-4783-ad46-1b92b15b8111"). InnerVolumeSpecName "kube-api-access-q2jqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:16:01.754974 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.754921    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3ba79e-77fd-4783-ad46-1b92b15b8111-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae3ba79e-77fd-4783-ad46-1b92b15b8111" (UID: "ae3ba79e-77fd-4783-ad46-1b92b15b8111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:16:01.853432 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.853405    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jqs\" (UniqueName: \"kubernetes.io/projected/ae3ba79e-77fd-4783-ad46-1b92b15b8111-kube-api-access-q2jqs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:16:01.853432 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.853430    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3ba79e-77fd-4783-ad46-1b92b15b8111-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:16:01.968039 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.968010    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerID="c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5" exitCode=0
Sep 18 00:16:01.968142 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.968058    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"
Sep 18 00:16:01.968142 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.968089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" event={"ID":"ae3ba79e-77fd-4783-ad46-1b92b15b8111","Type":"ContainerDied","Data":"c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5"}
Sep 18 00:16:01.968223 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.968141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8" event={"ID":"ae3ba79e-77fd-4783-ad46-1b92b15b8111","Type":"ContainerDied","Data":"7d60bb53ae4e2abf3df18bc0c4bb214c51ee8c9b1396d4b916f44af6209ffc48"}
Sep 18 00:16:01.968223 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.968167    2566 scope.go:117] "RemoveContainer" containerID="c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5"
Sep 18 00:16:01.979016 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.978996    2566 scope.go:117] "RemoveContainer" containerID="c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5"
Sep 18 00:16:01.979326 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:16:01.979306    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5\": container with ID starting with c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5 not found: ID does not exist" containerID="c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5"
Sep 18 00:16:01.979416 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.979333    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5"} err="failed to get container status \"c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5\": rpc error: code = NotFound desc = could not find container \"c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5\": container with ID starting with c3e6a6a0a9302c970db71d1ba9178a082ff0e3fe87e26fd45ea5411d36ede9c5 not found: ID does not exist"
Sep 18 00:16:01.994777 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.994749    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"]
Sep 18 00:16:01.998489 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:01.998466    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schectnn8"]
Sep 18 00:16:02.014045 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:02.013993    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" path="/var/lib/kubelet/pods/ae3ba79e-77fd-4783-ad46-1b92b15b8111/volumes"
Sep 18 00:16:24.999246 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999194    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"]
Sep 18 00:16:24.999807 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999737    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main"
Sep 18 00:16:24.999807 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999761    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main"
Sep 18 00:16:24.999807 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999793    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="storage-initializer"
Sep 18 00:16:24.999807 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999802    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="storage-initializer"
Sep 18 00:16:25.000045 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999817    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main"
Sep 18 00:16:25.000045 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999826    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main"
Sep 18 00:16:25.000045 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999926    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1333d310-cca3-417a-8fea-ef8a51ebc428" containerName="main"
Sep 18 00:16:25.000045 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:24.999943    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3ba79e-77fd-4783-ad46-1b92b15b8111" containerName="main"
Sep 18 00:16:25.004359 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.004333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.008530 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.008495    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 00:16:25.008686 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.008537    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 00:16:25.008686 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.008638    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-9kgdb\""
Sep 18 00:16:25.008686 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.008649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 00:16:25.011309 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.011281    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"]
Sep 18 00:16:25.126302 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.126276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0788890f-2a76-4096-8b39-445acb5bd91a-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.126427 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.126309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.126427 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.126341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4g2f\" (UniqueName: \"kubernetes.io/projected/0788890f-2a76-4096-8b39-445acb5bd91a-kube-api-access-q4g2f\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.126427 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.126387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.126550 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.126431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.126550 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.126460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227118 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0788890f-2a76-4096-8b39-445acb5bd91a-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227236 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227236 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4g2f\" (UniqueName: \"kubernetes.io/projected/0788890f-2a76-4096-8b39-445acb5bd91a-kube-api-access-q4g2f\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227353 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227504 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227719 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.227809 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.227713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.229555 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.229533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.229732 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.229713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0788890f-2a76-4096-8b39-445acb5bd91a-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.235143 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.235122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4g2f\" (UniqueName: \"kubernetes.io/projected/0788890f-2a76-4096-8b39-445acb5bd91a-kube-api-access-q4g2f\") pod \"scheduler-inline-config-test-kserve-8655bc757-6qpcs\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.317626 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.317535    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:25.587217 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:25.587189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"]
Sep 18 00:16:25.589818 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:16:25.589787    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0788890f_2a76_4096_8b39_445acb5bd91a.slice/crio-0ed0cf6c514601da0c0e7b1d6353b74d15899a46c12e4885adb44a05a5f5e61e WatchSource:0}: Error finding container 0ed0cf6c514601da0c0e7b1d6353b74d15899a46c12e4885adb44a05a5f5e61e: Status 404 returned error can't find the container with id 0ed0cf6c514601da0c0e7b1d6353b74d15899a46c12e4885adb44a05a5f5e61e
Sep 18 00:16:26.083381 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:26.083348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" event={"ID":"0788890f-2a76-4096-8b39-445acb5bd91a","Type":"ContainerStarted","Data":"ede045ac76cf6c92527aa0d685a3ddd567da4361cce81cb8fc4b445dcbaad1be"}
Sep 18 00:16:26.083819 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:26.083402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" event={"ID":"0788890f-2a76-4096-8b39-445acb5bd91a","Type":"ContainerStarted","Data":"0ed0cf6c514601da0c0e7b1d6353b74d15899a46c12e4885adb44a05a5f5e61e"}
Sep 18 00:16:26.100923 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:26.100878    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" podStartSLOduration=2.100865847 podStartE2EDuration="2.100865847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:16:26.099200784 +0000 UTC m=+3946.551032294" watchObservedRunningTime="2026-09-18 00:16:26.100865847 +0000 UTC m=+3946.552697334"
Sep 18 00:16:26.317644 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:26.317615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:35.318711 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:35.318616    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:35.331316 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:35.331280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:36.140966 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:36.140930    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:38.502094 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.502057    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"]
Sep 18 00:16:38.505256 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.505236    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:38.507949 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.507925    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 00:16:38.508061 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.507953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-lj9wl\""
Sep 18 00:16:38.509112 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.509090    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 00:16:38.509277 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.509243    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 00:16:38.509396 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.509250    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-65tb6\""
Sep 18 00:16:38.515597 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.515556    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"]
Sep 18 00:16:38.632105 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.632069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c23f14-6841-4ea2-b155-70713f50a2fb-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:38.632273 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.632128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pnm\" (UniqueName: \"kubernetes.io/projected/81c23f14-6841-4ea2-b155-70713f50a2fb-kube-api-access-b6pnm\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:38.732861 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.732830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pnm\" (UniqueName: \"kubernetes.io/projected/81c23f14-6841-4ea2-b155-70713f50a2fb-kube-api-access-b6pnm\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:38.732989 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.732888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c23f14-6841-4ea2-b155-70713f50a2fb-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:38.735528 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.735503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c23f14-6841-4ea2-b155-70713f50a2fb-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:38.741305 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.741280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pnm\" (UniqueName: \"kubernetes.io/projected/81c23f14-6841-4ea2-b155-70713f50a2fb-kube-api-access-b6pnm\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:38.818906 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:38.818846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:39.093750 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:39.093721    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"]
Sep 18 00:16:39.096601 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:16:39.096561    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c23f14_6841_4ea2_b155_70713f50a2fb.slice/crio-aae44bbd991ab525936f9ede93590f58c833d11e766e1aa03340fafa0430ccd7 WatchSource:0}: Error finding container aae44bbd991ab525936f9ede93590f58c833d11e766e1aa03340fafa0430ccd7: Status 404 returned error can't find the container with id aae44bbd991ab525936f9ede93590f58c833d11e766e1aa03340fafa0430ccd7
Sep 18 00:16:39.146253 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:39.146221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" event={"ID":"81c23f14-6841-4ea2-b155-70713f50a2fb","Type":"ContainerStarted","Data":"aae44bbd991ab525936f9ede93590f58c833d11e766e1aa03340fafa0430ccd7"}
Sep 18 00:16:40.152474 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:40.152436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" event={"ID":"81c23f14-6841-4ea2-b155-70713f50a2fb","Type":"ContainerStarted","Data":"d758a8bb375a6688555a4a30ec12dd407dd8950244c2cc2df57543fded208e67"}
Sep 18 00:16:40.152892 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:40.152570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:16:40.172614 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:40.172556    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" podStartSLOduration=2.172540915 podStartE2EDuration="2.172540915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:16:40.169832167 +0000 UTC m=+3960.621663666" watchObservedRunningTime="2026-09-18 00:16:40.172540915 +0000 UTC m=+3960.624372402"
Sep 18 00:16:58.938885 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:58.938843    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"]
Sep 18 00:16:58.950339 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:16:58.950305    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 00:16:58.950339 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:16:58.950339 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 18 00:16:58.950339 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="ede045ac76cf6c92527aa0d685a3ddd567da4361cce81cb8fc4b445dcbaad1be" cmd=["/bin/sleep","15"]
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:58.950353    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" message="" err=<
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:16:58.950368    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" podUID="0788890f-2a76-4096-8b39-445acb5bd91a" containerName="main" containerID="cri-o://ede045ac76cf6c92527aa0d685a3ddd567da4361cce81cb8fc4b445dcbaad1be"
Sep 18 00:16:58.950720 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:58.950409    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" podUID="0788890f-2a76-4096-8b39-445acb5bd91a" containerName="main" containerID="cri-o://ede045ac76cf6c92527aa0d685a3ddd567da4361cce81cb8fc4b445dcbaad1be" gracePeriod=60
Sep 18 00:16:59.129985 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.129940    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" podUID="0788890f-2a76-4096-8b39-445acb5bd91a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 00:16:59.239988 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.239953    2566 generic.go:356] "Generic (PLEG): container finished" podID="0788890f-2a76-4096-8b39-445acb5bd91a" containerID="ede045ac76cf6c92527aa0d685a3ddd567da4361cce81cb8fc4b445dcbaad1be" exitCode=0
Sep 18 00:16:59.240156 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.240030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" event={"ID":"0788890f-2a76-4096-8b39-445acb5bd91a","Type":"ContainerDied","Data":"ede045ac76cf6c92527aa0d685a3ddd567da4361cce81cb8fc4b445dcbaad1be"}
Sep 18 00:16:59.291061 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.291031    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:16:59.388876 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.388845    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-tmp-dir\") pod \"0788890f-2a76-4096-8b39-445acb5bd91a\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") "
Sep 18 00:16:59.388998 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.388887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0788890f-2a76-4096-8b39-445acb5bd91a-tls-certs\") pod \"0788890f-2a76-4096-8b39-445acb5bd91a\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") "
Sep 18 00:16:59.388998 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.388909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-home\") pod \"0788890f-2a76-4096-8b39-445acb5bd91a\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") "
Sep 18 00:16:59.389075 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389055    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-model-cache\") pod \"0788890f-2a76-4096-8b39-445acb5bd91a\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") "
Sep 18 00:16:59.389116 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389089    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4g2f\" (UniqueName: \"kubernetes.io/projected/0788890f-2a76-4096-8b39-445acb5bd91a-kube-api-access-q4g2f\") pod \"0788890f-2a76-4096-8b39-445acb5bd91a\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") "
Sep 18 00:16:59.389165 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-dshm\") pod \"0788890f-2a76-4096-8b39-445acb5bd91a\" (UID: \"0788890f-2a76-4096-8b39-445acb5bd91a\") "
Sep 18 00:16:59.389210 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0788890f-2a76-4096-8b39-445acb5bd91a" (UID: "0788890f-2a76-4096-8b39-445acb5bd91a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:59.389210 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389168    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-home" (OuterVolumeSpecName: "home") pod "0788890f-2a76-4096-8b39-445acb5bd91a" (UID: "0788890f-2a76-4096-8b39-445acb5bd91a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:59.389316 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0788890f-2a76-4096-8b39-445acb5bd91a" (UID: "0788890f-2a76-4096-8b39-445acb5bd91a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:59.389409 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389391    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:16:59.389409 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389407    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:16:59.389610 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.389415    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:16:59.391204 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.391177    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-dshm" (OuterVolumeSpecName: "dshm") pod "0788890f-2a76-4096-8b39-445acb5bd91a" (UID: "0788890f-2a76-4096-8b39-445acb5bd91a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:59.391285 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.391270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0788890f-2a76-4096-8b39-445acb5bd91a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0788890f-2a76-4096-8b39-445acb5bd91a" (UID: "0788890f-2a76-4096-8b39-445acb5bd91a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:16:59.391331 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.391273    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0788890f-2a76-4096-8b39-445acb5bd91a-kube-api-access-q4g2f" (OuterVolumeSpecName: "kube-api-access-q4g2f") pod "0788890f-2a76-4096-8b39-445acb5bd91a" (UID: "0788890f-2a76-4096-8b39-445acb5bd91a"). InnerVolumeSpecName "kube-api-access-q4g2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:16:59.490462 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.490441    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4g2f\" (UniqueName: \"kubernetes.io/projected/0788890f-2a76-4096-8b39-445acb5bd91a-kube-api-access-q4g2f\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:16:59.490462 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.490461    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0788890f-2a76-4096-8b39-445acb5bd91a-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:16:59.490607 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:16:59.490472    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0788890f-2a76-4096-8b39-445acb5bd91a-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:17:00.247330 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:00.247304    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"
Sep 18 00:17:00.247781 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:00.247306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs" event={"ID":"0788890f-2a76-4096-8b39-445acb5bd91a","Type":"ContainerDied","Data":"0ed0cf6c514601da0c0e7b1d6353b74d15899a46c12e4885adb44a05a5f5e61e"}
Sep 18 00:17:00.247781 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:00.247460    2566 scope.go:117] "RemoveContainer" containerID="ede045ac76cf6c92527aa0d685a3ddd567da4361cce81cb8fc4b445dcbaad1be"
Sep 18 00:17:00.263797 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:00.263774    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"]
Sep 18 00:17:00.266479 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:00.266456    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6qpcs"]
Sep 18 00:17:02.014722 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:02.014682    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0788890f-2a76-4096-8b39-445acb5bd91a" path="/var/lib/kubelet/pods/0788890f-2a76-4096-8b39-445acb5bd91a/volumes"
Sep 18 00:17:11.159123 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:11.159092    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:17:12.788435 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.788397    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"]
Sep 18 00:17:12.788834 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.788748    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0788890f-2a76-4096-8b39-445acb5bd91a" containerName="main"
Sep 18 00:17:12.788834 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.788759    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0788890f-2a76-4096-8b39-445acb5bd91a" containerName="main"
Sep 18 00:17:12.788834 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.788818    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0788890f-2a76-4096-8b39-445acb5bd91a" containerName="main"
Sep 18 00:17:12.793149 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.793128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.803407 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.803382    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"]
Sep 18 00:17:12.884368 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.884331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.884368 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.884371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77s8\" (UniqueName: \"kubernetes.io/projected/3ff7c8b3-131a-419d-b87d-2f7ab4613698-kube-api-access-t77s8\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.884549 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.884444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.884549 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.884467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.884549 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.884488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.884549 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.884523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.984842 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.984817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77s8\" (UniqueName: \"kubernetes.io/projected/3ff7c8b3-131a-419d-b87d-2f7ab4613698-kube-api-access-t77s8\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.984954 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.984874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.984954 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.984895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.984954 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.984914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.984954 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.984937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.985117 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.984965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.985356 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.985335    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.985426 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.985384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.985505 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.985481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.987311 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.987286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.987505 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.987490    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:12.991935 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:12.991915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77s8\" (UniqueName: \"kubernetes.io/projected/3ff7c8b3-131a-419d-b87d-2f7ab4613698-kube-api-access-t77s8\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c9cfv\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:13.105483 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:13.105424    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:13.374357 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:13.374331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"]
Sep 18 00:17:13.376355 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:17:13.376326    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff7c8b3_131a_419d_b87d_2f7ab4613698.slice/crio-37d4ca226aca4493b56ff3738e42322c8b42b9819fc0fd1ccb51660d7a41a705 WatchSource:0}: Error finding container 37d4ca226aca4493b56ff3738e42322c8b42b9819fc0fd1ccb51660d7a41a705: Status 404 returned error can't find the container with id 37d4ca226aca4493b56ff3738e42322c8b42b9819fc0fd1ccb51660d7a41a705
Sep 18 00:17:14.318184 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:14.318131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" event={"ID":"3ff7c8b3-131a-419d-b87d-2f7ab4613698","Type":"ContainerStarted","Data":"0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064"}
Sep 18 00:17:14.318710 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:14.318203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" event={"ID":"3ff7c8b3-131a-419d-b87d-2f7ab4613698","Type":"ContainerStarted","Data":"37d4ca226aca4493b56ff3738e42322c8b42b9819fc0fd1ccb51660d7a41a705"}
Sep 18 00:17:14.337345 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:14.337291    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" podStartSLOduration=2.337273181 podStartE2EDuration="2.337273181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:17:14.334781594 +0000 UTC m=+3994.786613088" watchObservedRunningTime="2026-09-18 00:17:14.337273181 +0000 UTC m=+3994.789104669"
Sep 18 00:17:15.106594 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:15.106548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:23.106324 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:23.106293    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:23.118931 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:23.118908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:23.372759 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:23.372688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:33.461725 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.461688    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"]
Sep 18 00:17:33.464201 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.464169    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"]
Sep 18 00:17:33.474441 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:17:33.474401    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 00:17:33.474441 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:17:33.474441 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 18 00:17:33.474441 ip-10-0-134-66 kubenswrapper[2566]:  > containerID="0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064" cmd=["/bin/sleep","15"]
Sep 18 00:17:33.474838 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.474810    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" message="" err=<
Sep 18 00:17:33.474838 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:17:33.474838 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 18 00:17:33.474838 ip-10-0-134-66 kubenswrapper[2566]:  >
Sep 18 00:17:33.475077 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:17:33.474840    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 00:17:33.475077 ip-10-0-134-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:17:33.475077 ip-10-0-134-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 18 00:17:33.475077 ip-10-0-134-66 kubenswrapper[2566]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" podUID="3ff7c8b3-131a-419d-b87d-2f7ab4613698" containerName="main" containerID="cri-o://0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064"
Sep 18 00:17:33.475077 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.474912    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" podUID="3ff7c8b3-131a-419d-b87d-2f7ab4613698" containerName="main" containerID="cri-o://0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064" gracePeriod=60
Sep 18 00:17:33.824174 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.824148    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:33.831937 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.831915    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-dshm\") pod \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") "
Sep 18 00:17:33.832055 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.831951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tmp-dir\") pod \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") "
Sep 18 00:17:33.832055 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.831974    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tls-certs\") pod \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") "
Sep 18 00:17:33.832055 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t77s8\" (UniqueName: \"kubernetes.io/projected/3ff7c8b3-131a-419d-b87d-2f7ab4613698-kube-api-access-t77s8\") pod \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") "
Sep 18 00:17:33.832055 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-model-cache\") pod \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") "
Sep 18 00:17:33.832285 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-home\") pod \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\" (UID: \"3ff7c8b3-131a-419d-b87d-2f7ab4613698\") "
Sep 18 00:17:33.832285 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ff7c8b3-131a-419d-b87d-2f7ab4613698" (UID: "3ff7c8b3-131a-419d-b87d-2f7ab4613698"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:33.832405 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ff7c8b3-131a-419d-b87d-2f7ab4613698" (UID: "3ff7c8b3-131a-419d-b87d-2f7ab4613698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:33.832405 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-home" (OuterVolumeSpecName: "home") pod "3ff7c8b3-131a-419d-b87d-2f7ab4613698" (UID: "3ff7c8b3-131a-419d-b87d-2f7ab4613698"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:33.832405 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832355    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-model-cache\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:17:33.832405 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.832381    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tmp-dir\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:17:33.834190 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.834167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-dshm" (OuterVolumeSpecName: "dshm") pod "3ff7c8b3-131a-419d-b87d-2f7ab4613698" (UID: "3ff7c8b3-131a-419d-b87d-2f7ab4613698"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:33.834309 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.834287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff7c8b3-131a-419d-b87d-2f7ab4613698-kube-api-access-t77s8" (OuterVolumeSpecName: "kube-api-access-t77s8") pod "3ff7c8b3-131a-419d-b87d-2f7ab4613698" (UID: "3ff7c8b3-131a-419d-b87d-2f7ab4613698"). InnerVolumeSpecName "kube-api-access-t77s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:17:33.834376 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.834304    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ff7c8b3-131a-419d-b87d-2f7ab4613698" (UID: "3ff7c8b3-131a-419d-b87d-2f7ab4613698"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:17:33.933006 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.932977    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77s8\" (UniqueName: \"kubernetes.io/projected/3ff7c8b3-131a-419d-b87d-2f7ab4613698-kube-api-access-t77s8\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:17:33.933006 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.933002    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-home\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:17:33.933006 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.933013    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff7c8b3-131a-419d-b87d-2f7ab4613698-dshm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:17:33.933202 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:33.933021    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff7c8b3-131a-419d-b87d-2f7ab4613698-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:17:34.418680 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.418561    2566 generic.go:356] "Generic (PLEG): container finished" podID="3ff7c8b3-131a-419d-b87d-2f7ab4613698" containerID="0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064" exitCode=0
Sep 18 00:17:34.418680 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.418618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" event={"ID":"3ff7c8b3-131a-419d-b87d-2f7ab4613698","Type":"ContainerDied","Data":"0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064"}
Sep 18 00:17:34.418680 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.418650    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"
Sep 18 00:17:34.418918 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.418692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv" event={"ID":"3ff7c8b3-131a-419d-b87d-2f7ab4613698","Type":"ContainerDied","Data":"37d4ca226aca4493b56ff3738e42322c8b42b9819fc0fd1ccb51660d7a41a705"}
Sep 18 00:17:34.418918 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.418719    2566 scope.go:117] "RemoveContainer" containerID="0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064"
Sep 18 00:17:34.428383 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.428364    2566 scope.go:117] "RemoveContainer" containerID="0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064"
Sep 18 00:17:34.428737 ip-10-0-134-66 kubenswrapper[2566]: E0918 00:17:34.428713    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064\": container with ID starting with 0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064 not found: ID does not exist" containerID="0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064"
Sep 18 00:17:34.428830 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.428745    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064"} err="failed to get container status \"0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064\": rpc error: code = NotFound desc = could not find container \"0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064\": container with ID starting with 0005a53cf078260077b9d20f14b7d4671bece1822f3e0e398ea200de26707064 not found: ID does not exist"
Sep 18 00:17:34.435437 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.435408    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"]
Sep 18 00:17:34.438977 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:34.438949    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c9cfv"]
Sep 18 00:17:36.014986 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:36.014943    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff7c8b3-131a-419d-b87d-2f7ab4613698" path="/var/lib/kubelet/pods/3ff7c8b3-131a-419d-b87d-2f7ab4613698/volumes"
Sep 18 00:17:41.159320 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:41.159278    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:17:48.486146 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:48.486091    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main" containerID="cri-o://d758a8bb375a6688555a4a30ec12dd407dd8950244c2cc2df57543fded208e67" gracePeriod=45
Sep 18 00:17:51.159528 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:51.159483    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:17:59.143258 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.143226    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"]
Sep 18 00:17:59.143695 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.143553    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff7c8b3-131a-419d-b87d-2f7ab4613698" containerName="main"
Sep 18 00:17:59.143695 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.143564    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff7c8b3-131a-419d-b87d-2f7ab4613698" containerName="main"
Sep 18 00:17:59.143813 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.143722    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff7c8b3-131a-419d-b87d-2f7ab4613698" containerName="main"
Sep 18 00:17:59.148282 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.148263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.150936 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.150912    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 00:17:59.151070 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.150940    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-zczm6\""
Sep 18 00:17:59.152385 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.152363    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-lwnzq\""
Sep 18 00:17:59.152718 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.152427    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 00:17:59.152855 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.152489    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 00:17:59.155663 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.155611    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"]
Sep 18 00:17:59.215979 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.215957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jcxv\" (UniqueName: \"kubernetes.io/projected/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-kube-api-access-9jcxv\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.216085 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.215987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.316384 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.316362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jcxv\" (UniqueName: \"kubernetes.io/projected/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-kube-api-access-9jcxv\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.316482 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.316392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.318942 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.318920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.325504 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.325484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jcxv\" (UniqueName: \"kubernetes.io/projected/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-kube-api-access-9jcxv\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.460801 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.460742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:17:59.729832 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:17:59.729804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"]
Sep 18 00:17:59.731965 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:17:59.731929    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97fbb4c_c90a_4117_8ce8_4ba8b65e93d0.slice/crio-a649bfef55d8c2d410e721b625d9bcec34d9c7aa2f502ed48a470b62cc5a7750 WatchSource:0}: Error finding container a649bfef55d8c2d410e721b625d9bcec34d9c7aa2f502ed48a470b62cc5a7750: Status 404 returned error can't find the container with id a649bfef55d8c2d410e721b625d9bcec34d9c7aa2f502ed48a470b62cc5a7750
Sep 18 00:18:00.552195 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:00.552157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" event={"ID":"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0","Type":"ContainerStarted","Data":"91cbed5ee70f4d4679a7a25bbd353566a1b1f05f2d78f04c68fe881bf1cb4f7e"}
Sep 18 00:18:00.552654 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:00.552237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" event={"ID":"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0","Type":"ContainerStarted","Data":"a649bfef55d8c2d410e721b625d9bcec34d9c7aa2f502ed48a470b62cc5a7750"}
Sep 18 00:18:00.552654 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:00.552279    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:18:00.568253 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:00.568209    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" podStartSLOduration=1.5681946839999998 podStartE2EDuration="1.568194684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:18:00.567387711 +0000 UTC m=+4041.019219199" watchObservedRunningTime="2026-09-18 00:18:00.568194684 +0000 UTC m=+4041.020026172"
Sep 18 00:18:01.159236 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:01.159196    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:18:01.159415 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:01.159317    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:18:11.159223 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:11.159135    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:18:18.648295 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.648257    2566 generic.go:356] "Generic (PLEG): container finished" podID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerID="d758a8bb375a6688555a4a30ec12dd407dd8950244c2cc2df57543fded208e67" exitCode=0
Sep 18 00:18:18.648707 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.648339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" event={"ID":"81c23f14-6841-4ea2-b155-70713f50a2fb","Type":"ContainerDied","Data":"d758a8bb375a6688555a4a30ec12dd407dd8950244c2cc2df57543fded208e67"}
Sep 18 00:18:18.725135 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.725106    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:18:18.872762 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.872693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c23f14-6841-4ea2-b155-70713f50a2fb-tls-certs\") pod \"81c23f14-6841-4ea2-b155-70713f50a2fb\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") "
Sep 18 00:18:18.872762 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.872735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6pnm\" (UniqueName: \"kubernetes.io/projected/81c23f14-6841-4ea2-b155-70713f50a2fb-kube-api-access-b6pnm\") pod \"81c23f14-6841-4ea2-b155-70713f50a2fb\" (UID: \"81c23f14-6841-4ea2-b155-70713f50a2fb\") "
Sep 18 00:18:18.874798 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.874766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c23f14-6841-4ea2-b155-70713f50a2fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81c23f14-6841-4ea2-b155-70713f50a2fb" (UID: "81c23f14-6841-4ea2-b155-70713f50a2fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:18:18.874918 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.874837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c23f14-6841-4ea2-b155-70713f50a2fb-kube-api-access-b6pnm" (OuterVolumeSpecName: "kube-api-access-b6pnm") pod "81c23f14-6841-4ea2-b155-70713f50a2fb" (UID: "81c23f14-6841-4ea2-b155-70713f50a2fb"). InnerVolumeSpecName "kube-api-access-b6pnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:18:18.973379 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.973348    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c23f14-6841-4ea2-b155-70713f50a2fb-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:18:18.973379 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:18.973375    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pnm\" (UniqueName: \"kubernetes.io/projected/81c23f14-6841-4ea2-b155-70713f50a2fb-kube-api-access-b6pnm\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:18:19.654723 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:19.654692    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"
Sep 18 00:18:19.655184 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:19.654680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs" event={"ID":"81c23f14-6841-4ea2-b155-70713f50a2fb","Type":"ContainerDied","Data":"aae44bbd991ab525936f9ede93590f58c833d11e766e1aa03340fafa0430ccd7"}
Sep 18 00:18:19.655184 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:19.654851    2566 scope.go:117] "RemoveContainer" containerID="d758a8bb375a6688555a4a30ec12dd407dd8950244c2cc2df57543fded208e67"
Sep 18 00:18:19.675024 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:19.674998    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"]
Sep 18 00:18:19.678722 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:19.678699    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-bk4zs"]
Sep 18 00:18:20.014930 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:20.014898    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" path="/var/lib/kubelet/pods/81c23f14-6841-4ea2-b155-70713f50a2fb/volumes"
Sep 18 00:18:31.559539 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:31.559509    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:18:46.804080 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:46.804047    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"]
Sep 18 00:18:51.560084 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:18:51.560038    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:19:01.559870 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:01.559830    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:19:01.824526 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:01.824426    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main" containerID="cri-o://91cbed5ee70f4d4679a7a25bbd353566a1b1f05f2d78f04c68fe881bf1cb4f7e" gracePeriod=45
Sep 18 00:19:11.559375 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:11.559322    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:19:11.559862 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:11.559473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:19:21.560113 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:21.560049    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:19:31.558979 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:31.558927    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 00:19:32.016392 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.016327    2566 generic.go:356] "Generic (PLEG): container finished" podID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerID="91cbed5ee70f4d4679a7a25bbd353566a1b1f05f2d78f04c68fe881bf1cb4f7e" exitCode=0
Sep 18 00:19:32.016556 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.016401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" event={"ID":"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0","Type":"ContainerDied","Data":"91cbed5ee70f4d4679a7a25bbd353566a1b1f05f2d78f04c68fe881bf1cb4f7e"}
Sep 18 00:19:32.080641 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.079019    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:19:32.213443 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.213378    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jcxv\" (UniqueName: \"kubernetes.io/projected/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-kube-api-access-9jcxv\") pod \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") "
Sep 18 00:19:32.213602 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.213457    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-tls-certs\") pod \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\" (UID: \"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0\") "
Sep 18 00:19:32.215691 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.215659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" (UID: "b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:19:32.215785 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.215659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-kube-api-access-9jcxv" (OuterVolumeSpecName: "kube-api-access-9jcxv") pod "b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" (UID: "b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0"). InnerVolumeSpecName "kube-api-access-9jcxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:19:32.314763 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.314737    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jcxv\" (UniqueName: \"kubernetes.io/projected/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-kube-api-access-9jcxv\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:19:32.314861 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:32.314765    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0-tls-certs\") on node \"ip-10-0-134-66.ec2.internal\" DevicePath \"\""
Sep 18 00:19:33.021987 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:33.021924    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"
Sep 18 00:19:33.021987 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:33.021956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx" event={"ID":"b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0","Type":"ContainerDied","Data":"a649bfef55d8c2d410e721b625d9bcec34d9c7aa2f502ed48a470b62cc5a7750"}
Sep 18 00:19:33.022413 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:33.022005    2566 scope.go:117] "RemoveContainer" containerID="91cbed5ee70f4d4679a7a25bbd353566a1b1f05f2d78f04c68fe881bf1cb4f7e"
Sep 18 00:19:33.044011 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:33.043985    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"]
Sep 18 00:19:33.047501 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:33.047479    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmvnx"]
Sep 18 00:19:34.013743 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:19:34.013708    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" path="/var/lib/kubelet/pods/b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0/volumes"
Sep 18 00:20:19.121081 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:19.121046    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94b6fc96d-x49tb_b697b71a-4297-4445-9f49-092a0217e8b8/router/0.log"
Sep 18 00:20:19.965135 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:19.965107    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94b6fc96d-x49tb_b697b71a-4297-4445-9f49-092a0217e8b8/router/0.log"
Sep 18 00:20:20.777684 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:20.777652    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-jrtvd_d99d8a01-0aaf-41ea-8d71-d51925c8d953/authorino/0.log"
Sep 18 00:20:20.821146 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:20.821122    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4q6c9_995f2cd8-7ab1-4ddd-a0ef-418d2f591ced/manager/0.log"
Sep 18 00:20:20.915354 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:20.915327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-qjk4h_532fa764-0bdc-43cd-9015-98e2da6879f4/manager/0.log"
Sep 18 00:20:29.404713 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:29.404671    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jkmmb_50534f9a-9cd4-4dfb-9e76-02e8f2247ed7/global-pull-secret-syncer/0.log"
Sep 18 00:20:29.531913 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:29.531880    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhsgn_3162a2e0-39fc-43da-97f6-18fd0932c183/konnectivity-agent/0.log"
Sep 18 00:20:29.638412 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:29.638383    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-66.ec2.internal_b015008b02def3f3ba52667fd2a6cf91/haproxy/0.log"
Sep 18 00:20:34.522428 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:34.522397    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-jrtvd_d99d8a01-0aaf-41ea-8d71-d51925c8d953/authorino/0.log"
Sep 18 00:20:34.574702 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:34.574658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4q6c9_995f2cd8-7ab1-4ddd-a0ef-418d2f591ced/manager/0.log"
Sep 18 00:20:34.691023 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:34.690996    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-qjk4h_532fa764-0bdc-43cd-9015-98e2da6879f4/manager/0.log"
Sep 18 00:20:36.332983 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.332890    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-zndfk_8d3df576-687c-4581-ad5a-09f859fd41c2/kube-state-metrics/0.log"
Sep 18 00:20:36.351891 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.351854    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-zndfk_8d3df576-687c-4581-ad5a-09f859fd41c2/kube-rbac-proxy-main/0.log"
Sep 18 00:20:36.372968 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.372943    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-zndfk_8d3df576-687c-4581-ad5a-09f859fd41c2/kube-rbac-proxy-self/0.log"
Sep 18 00:20:36.523774 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.523755    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpd8l_271151eb-ca8a-4d11-8b92-a1c3b30c5e31/node-exporter/0.log"
Sep 18 00:20:36.543154 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.543137    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpd8l_271151eb-ca8a-4d11-8b92-a1c3b30c5e31/kube-rbac-proxy/0.log"
Sep 18 00:20:36.562980 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.562963    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpd8l_271151eb-ca8a-4d11-8b92-a1c3b30c5e31/init-textfile/0.log"
Sep 18 00:20:36.750774 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.750751    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d983559-4241-422c-89b9-0dac7d6b6870/prometheus/0.log"
Sep 18 00:20:36.767072 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.767049    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d983559-4241-422c-89b9-0dac7d6b6870/config-reloader/0.log"
Sep 18 00:20:36.791566 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.791544    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d983559-4241-422c-89b9-0dac7d6b6870/thanos-sidecar/0.log"
Sep 18 00:20:36.813285 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.813261    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d983559-4241-422c-89b9-0dac7d6b6870/kube-rbac-proxy-web/0.log"
Sep 18 00:20:36.831803 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.831785    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d983559-4241-422c-89b9-0dac7d6b6870/kube-rbac-proxy/0.log"
Sep 18 00:20:36.853392 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.853330    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d983559-4241-422c-89b9-0dac7d6b6870/kube-rbac-proxy-thanos/0.log"
Sep 18 00:20:36.874721 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.874701    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d983559-4241-422c-89b9-0dac7d6b6870/init-config-reloader/0.log"
Sep 18 00:20:36.977556 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.977534    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76b5dd45c-7zf7w_ede67b21-37b8-48cc-9b38-c2d30dccbfa5/telemeter-client/0.log"
Sep 18 00:20:36.998759 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:36.998735    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76b5dd45c-7zf7w_ede67b21-37b8-48cc-9b38-c2d30dccbfa5/reload/0.log"
Sep 18 00:20:37.021140 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:37.021124    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76b5dd45c-7zf7w_ede67b21-37b8-48cc-9b38-c2d30dccbfa5/kube-rbac-proxy/0.log"
Sep 18 00:20:39.027720 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:39.027627    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:20:39.032098 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:39.032072    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/3.log"
Sep 18 00:20:40.143702 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.143667    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"]
Sep 18 00:20:40.144094 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.143958    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main"
Sep 18 00:20:40.144094 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.143968    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main"
Sep 18 00:20:40.144094 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.143981    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main"
Sep 18 00:20:40.144094 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.143987    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main"
Sep 18 00:20:40.144094 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.144062    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c23f14-6841-4ea2-b155-70713f50a2fb" containerName="main"
Sep 18 00:20:40.144094 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.144071    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97fbb4c-c90a-4117-8ce8-4ba8b65e93d0" containerName="main"
Sep 18 00:20:40.147413 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.147392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.150102 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.150086    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8545l\"/\"kube-root-ca.crt\""
Sep 18 00:20:40.151311 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.151292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8545l\"/\"openshift-service-ca.crt\""
Sep 18 00:20:40.151373 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.151298    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8545l\"/\"default-dockercfg-vqjqc\""
Sep 18 00:20:40.157018 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.156993    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"]
Sep 18 00:20:40.223498 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.223471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-proc\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.223627 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.223552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-podres\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.223627 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.223608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-lib-modules\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.223721 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.223626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svd2n\" (UniqueName: \"kubernetes.io/projected/7c058855-e9c7-45dc-a282-8c4c94031d41-kube-api-access-svd2n\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.223721 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.223656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-sys\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324555 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-podres\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324679 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-lib-modules\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324679 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svd2n\" (UniqueName: \"kubernetes.io/projected/7c058855-e9c7-45dc-a282-8c4c94031d41-kube-api-access-svd2n\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324679 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-sys\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324804 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-proc\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324804 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-podres\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324804 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-lib-modules\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324804 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-sys\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.324804 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.324798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c058855-e9c7-45dc-a282-8c4c94031d41-proc\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.333803 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.333778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svd2n\" (UniqueName: \"kubernetes.io/projected/7c058855-e9c7-45dc-a282-8c4c94031d41-kube-api-access-svd2n\") pod \"perf-node-gather-daemonset-th88k\" (UID: \"7c058855-e9c7-45dc-a282-8c4c94031d41\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.378860 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.378838    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:20:40.387021 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.386992    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:20:40.390651 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.390634    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-hzpdg_db1763ab-9b54-4c5a-81b6-f390e9d0859b/console-operator/2.log"
Sep 18 00:20:40.398339 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.398288    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:20:40.457778 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.457757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:40.728424 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:40.728395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"]
Sep 18 00:20:40.731134 ip-10-0-134-66 kubenswrapper[2566]: W0918 00:20:40.731105    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c058855_e9c7_45dc_a282_8c4c94031d41.slice/crio-65f0b23f2fc92c48b921d1e66ab703d987770eb03831d1c0f7c1c3d305a15112 WatchSource:0}: Error finding container 65f0b23f2fc92c48b921d1e66ab703d987770eb03831d1c0f7c1c3d305a15112: Status 404 returned error can't find the container with id 65f0b23f2fc92c48b921d1e66ab703d987770eb03831d1c0f7c1c3d305a15112
Sep 18 00:20:41.095043 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.095015    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzxvm_6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d/dns/0.log"
Sep 18 00:20:41.112617 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.112598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzxvm_6ec0d4f1-7308-4faa-aa7f-ce57dcc7690d/kube-rbac-proxy/0.log"
Sep 18 00:20:41.170525 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.170505    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gq5q5_938d9893-bdce-4bd2-8898-0ab74c7fc325/dns-node-resolver/0.log"
Sep 18 00:20:41.341380 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.341346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k" event={"ID":"7c058855-e9c7-45dc-a282-8c4c94031d41","Type":"ContainerStarted","Data":"5888e486c606ad032458877d742003149010ca4eb9dfe84da1c28aca0d1dd398"}
Sep 18 00:20:41.341511 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.341412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k" event={"ID":"7c058855-e9c7-45dc-a282-8c4c94031d41","Type":"ContainerStarted","Data":"65f0b23f2fc92c48b921d1e66ab703d987770eb03831d1c0f7c1c3d305a15112"}
Sep 18 00:20:41.341561 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.341527    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:41.357466 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.357391    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k" podStartSLOduration=1.35737963 podStartE2EDuration="1.35737963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:20:41.355192502 +0000 UTC m=+4201.807024045" watchObservedRunningTime="2026-09-18 00:20:41.35737963 +0000 UTC m=+4201.809211117"
Sep 18 00:20:41.725457 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:41.725371    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nvz6b_0be814cf-7b90-4d33-9f35-f73fdf631402/node-ca/0.log"
Sep 18 00:20:42.833587 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:42.833554    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94b6fc96d-x49tb_b697b71a-4297-4445-9f49-092a0217e8b8/router/0.log"
Sep 18 00:20:43.323840 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:43.323812    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjz88_e92de406-eab7-42ca-a693-611ad02f1377/serve-healthcheck-canary/0.log"
Sep 18 00:20:43.916006 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:43.915979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8cwbr_2812488d-c0cc-4067-9d1a-ac432eb0b7e1/kube-rbac-proxy/0.log"
Sep 18 00:20:43.933327 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:43.933304    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8cwbr_2812488d-c0cc-4067-9d1a-ac432eb0b7e1/exporter/0.log"
Sep 18 00:20:43.955481 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:43.955454    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8cwbr_2812488d-c0cc-4067-9d1a-ac432eb0b7e1/extractor/0.log"
Sep 18 00:20:46.831527 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:46.831484    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6944fcc464-565sl_46594425-5c87-4915-a01f-c58519b3fbd2/manager/0.log"
Sep 18 00:20:46.875778 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:46.875745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jgdk4_e153cda8-4e06-4e07-a0ba-39a562590ab9/openshift-lws-operator/0.log"
Sep 18 00:20:47.355474 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:47.355448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-th88k"
Sep 18 00:20:47.925055 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:47.925024    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-hk6j7_6f70af8b-63c6-4075-bbc7-bb909bdffb18/server/0.log"
Sep 18 00:20:48.425562 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:48.425533    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-spr8s_368951d8-63b6-4c36-a5ba-7c90ff58ee46/manager/0.log"
Sep 18 00:20:49.498132 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:49.498098    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94b6fc96d-x49tb_b697b71a-4297-4445-9f49-092a0217e8b8/router/0.log"
Sep 18 00:20:50.782563 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:50.782538    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94b6fc96d-x49tb_b697b71a-4297-4445-9f49-092a0217e8b8/router/0.log"
Sep 18 00:20:53.812795 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:53.812761    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-xwkcq_6591d17a-2227-4fc6-bcc5-cdc82be2605e/kube-storage-version-migrator-operator/1.log"
Sep 18 00:20:53.813615 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:53.813595    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-xwkcq_6591d17a-2227-4fc6-bcc5-cdc82be2605e/kube-storage-version-migrator-operator/0.log"
Sep 18 00:20:54.737462 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.737418    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7zhxm_2f99df5d-824d-4474-a81b-7cda0839ae43/kube-multus/0.log"
Sep 18 00:20:54.896231 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.896198    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvw78_f4e50a1d-15b6-42be-bcd4-641f72602ad1/kube-multus-additional-cni-plugins/0.log"
Sep 18 00:20:54.914078 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.914050    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvw78_f4e50a1d-15b6-42be-bcd4-641f72602ad1/egress-router-binary-copy/0.log"
Sep 18 00:20:54.930503 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.930481    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvw78_f4e50a1d-15b6-42be-bcd4-641f72602ad1/cni-plugins/0.log"
Sep 18 00:20:54.950819 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.950796    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvw78_f4e50a1d-15b6-42be-bcd4-641f72602ad1/bond-cni-plugin/0.log"
Sep 18 00:20:54.967137 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.967115    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvw78_f4e50a1d-15b6-42be-bcd4-641f72602ad1/routeoverride-cni/0.log"
Sep 18 00:20:54.983106 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.983085    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvw78_f4e50a1d-15b6-42be-bcd4-641f72602ad1/whereabouts-cni-bincopy/0.log"
Sep 18 00:20:54.999889 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:54.999838    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvw78_f4e50a1d-15b6-42be-bcd4-641f72602ad1/whereabouts-cni/0.log"
Sep 18 00:20:55.308922 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:55.308866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zd5lf_bdb8ea08-57db-405c-ac91-9de219efe014/network-metrics-daemon/0.log"
Sep 18 00:20:55.324922 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:55.324905    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zd5lf_bdb8ea08-57db-405c-ac91-9de219efe014/kube-rbac-proxy/0.log"
Sep 18 00:20:56.525949 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.525915    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-controller/0.log"
Sep 18 00:20:56.540299 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.540273    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/0.log"
Sep 18 00:20:56.540463 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.540308    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovn-acl-logging/1.log"
Sep 18 00:20:56.562360 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.562330    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/kube-rbac-proxy-node/0.log"
Sep 18 00:20:56.581297 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.581273    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 00:20:56.599755 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.599734    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/northd/0.log"
Sep 18 00:20:56.617557 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.617539    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/nbdb/0.log"
Sep 18 00:20:56.635323 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.635308    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/sbdb/0.log"
Sep 18 00:20:56.684895 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:56.684871    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rgw_e18d7e89-9ce2-4526-a68e-00f4ef8e82fe/ovnkube-controller/0.log"
Sep 18 00:20:57.687769 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:57.687737    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-kn5mt_bad94791-5a45-4383-bba5-64d08717cde3/check-endpoints/0.log"
Sep 18 00:20:57.711255 ip-10-0-134-66 kubenswrapper[2566]: I0918 00:20:57.711229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-kmt2w_58593c9f-7d98-433a-a4cd-9507eb36928d/network-check-target-container/0.log"