Sep 17 17:38:36.486306 ip-10-0-139-39 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:38:36.486319 ip-10-0-139-39 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:38:36.486328 ip-10-0-139-39 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:38:36.486716 ip-10-0-139-39 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:38:46.550819 ip-10-0-139-39 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:38:46.550839 ip-10-0-139-39 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot df178f5c1c9c4fccb4ba1c3ac9121147 --
Sep 17 17:40:23.568529 ip-10-0-139-39 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:40:23.864998 ip-10-0-139-39 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:23.864998 ip-10-0-139-39 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:40:23.864998 ip-10-0-139-39 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:23.864998 ip-10-0-139-39 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:40:23.864998 ip-10-0-139-39 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:23.866322 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.866184    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:40:23.869746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869726    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:23.869746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869745    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869750    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869755    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869758    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869762    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869765    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869768    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869772    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869775    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869778    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869780    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869783    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869786    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869789    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869791    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869794    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869797    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869800    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869802    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:23.869816 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869805    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869808    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869810    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869813    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869822    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869826    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869829    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869832    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869835    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869837    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869840    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869843    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869846    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869848    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869851    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869854    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869857    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869859    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869862    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869866    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:23.870291 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869868    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869871    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869873    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869876    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869880    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869885    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869888    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869891    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869894    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869897    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869899    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869902    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869905    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869908    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869911    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869914    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869917    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869920    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869922    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:23.870790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869925    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869928    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869930    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869933    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869935    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869938    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869941    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869943    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869945    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869948    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869951    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869953    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869957    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869960    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869963    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869965    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869968    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869972    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869974    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869977    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:23.871270 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869979    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869982    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869984    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869987    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869990    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869994    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.869998    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870003    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870006    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870009    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870012    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870014    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870017    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870020    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870023    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870026    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870030    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870033    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870036    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:23.871790 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870038    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870809    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870818    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870822    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870825    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870830    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870834    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870838    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870841    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870844    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870847    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870850    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870853    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870856    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870859    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870861    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870864    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870866    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870869    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:23.872266 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870872    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870875    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870878    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870881    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870883    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870886    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870888    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870891    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870894    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870896    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870899    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870901    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870904    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870906    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870909    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870912    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870914    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870917    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870919    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870922    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:23.872742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870925    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870928    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870931    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870934    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870936    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870939    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870942    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870944    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870947    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870949    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870952    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870955    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870957    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870960    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870963    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870966    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870968    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870971    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870973    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870976    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:23.873249 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870979    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870981    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870984    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870987    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870989    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870992    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870994    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870997    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.870999    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871002    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871005    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871007    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871010    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871013    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871016    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871018    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871021    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871023    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871026    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:23.873746 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871029    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871032    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871035    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871038    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871041    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871043    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871046    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871048    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871051    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871054    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871056    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871058    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871061    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871064    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871066    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871069    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871071    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871075    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871078    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871097    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:23.874230 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871100    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871103    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871524    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871529    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871532    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871535    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871539    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871542    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871544    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871547    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871549    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871553    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871555    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871559    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871562    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871565    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871568    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871571    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871573    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871576    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:23.874722 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871580    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871585    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871589    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871592    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871596    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871599    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871602    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871605    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871608    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871611    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871614    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871617    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871620    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871622    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871625    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871627    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871630    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871632    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871635    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871639    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:23.875238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871642    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871645    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871648    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871650    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871653    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871656    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871659    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871662    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871664    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871667    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871670    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871673    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871675    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871678    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871680    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871685    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871689    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871692    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871695    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:23.875721 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871698    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871701    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871704    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871707    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871710    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871713    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871716    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871718    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871721    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871724    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871727    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871729    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871732    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871736    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871739    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871742    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871744    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871747    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871750    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871753    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:23.876201 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871757    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871759    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871762    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871765    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871767    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871770    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871773    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871775    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871778    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871780    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871783    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871785    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871788    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871791    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871794    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871797    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871799    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871802    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871804    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871807    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:23.876727 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871810    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.871812    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871903    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871951    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871958    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871964    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871971    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871975    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871980    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871985    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871989    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871992    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.871996    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872000    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872003    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872007    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872010    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872013    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872016    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872019    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872027    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872031    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872034    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872038    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:40:23.877224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872042    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872046    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872049    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872052    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872056    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872059    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872063    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872066    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872069    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872073    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872077    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872080    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872096    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872099    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872102    2580 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872113    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872120    2580 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872123    2580 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872129    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872133    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872136    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872140    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872143    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872147    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872150    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:40:23.877806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872152    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872155    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872158    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872162    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872164    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872167    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872170    2580 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872174    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872178    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872181    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872184    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872187    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872193    2580 flags.go:64] FLAG: --help="false"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872196    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-139-39.ec2.internal"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872199    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872202    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872205    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872209    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872212    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872215    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872218    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872221    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872224    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:40:23.878430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872235    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872239    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872244    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872247    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872250    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872253    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872256    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872259    2580 flags.go:64] FLAG: --lock-file=""
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872262    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872265    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872268    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872275    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872278    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872281    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872284    2580 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872287    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872291    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872295    2580 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872298    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872303    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872307    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872313    2580 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872316    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872319    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872322    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:40:23.879023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872325    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872328    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872331    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872335    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872343    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872346    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872350    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872353    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872362    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872369    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872373    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872377    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872380    2580 flags.go:64] FLAG: --port="10250"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872383    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872386    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0342a15427309d650"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872389    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872392    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872395    2580 flags.go:64] FLAG: --register-node="true"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872398    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872402    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872405    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872408    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872411    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872415    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872418    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:40:23.880067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872421    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872424    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872427    2580 flags.go:64] FLAG: --runonce="false"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872431    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872435    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872438    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872441    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872445    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872448    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872451    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872454    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872457    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872460    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872464    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872467    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872470    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872479    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872483    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872488    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872491    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872494    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872499    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872502    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872505    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872508    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:40:23.881313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872511    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872514    2580 flags.go:64] FLAG: --v="2"
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872518    2580 flags.go:64] FLAG: --version="false"
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872522    2580 flags.go:64] FLAG: --vmodule=""
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872526    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.872530    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872685    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872691    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872694    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872697    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872702    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872705    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872708    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872712    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872714    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872717    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872720    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872723    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872726    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872728    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872731    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:23.882332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872734    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872737    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872740    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872745    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872748    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872751    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872754    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872757    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872760    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872762    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872765    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872769    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872773    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872776    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872779    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872781    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872784    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872786    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872789    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:23.883232 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872806    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872810    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872815    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872819    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872822    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872825    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872828    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872831    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872833    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872836    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872840    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872843    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872845    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872848    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872851    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872853    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872858    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872862    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872864    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872867    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:23.884074 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872870    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872873    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872876    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872878    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872881    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872884    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872886    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872889    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872892    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872895    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872899    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872903    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872906    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872909    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872914    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872916    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872919    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872922    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872925    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:23.884794 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872928    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872931    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872933    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872936    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872938    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872941    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872944    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872946    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872949    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872953    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872956    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872959    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872962    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872965    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872968    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872970    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872973    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872976    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872978    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872981    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:23.885391 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872984    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872986    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872989    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872992    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.872996    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.873000    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.873529    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.882126    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.882154    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.882806    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.882821    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882937    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882946    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882951    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:23.886019 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882955    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882960    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882965    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882970    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882974    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882979    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882983    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882988    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882992    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.882996    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883000    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883004    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883009    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883013    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883017    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883022    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883026    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883030    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883035    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883039    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:23.886630 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883044    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883049    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883054    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883058    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883063    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883067    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883099    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883105    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883109    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883112    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883116    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883120    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883124    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883127    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883131    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883135    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883145    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883148    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883152    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:23.887488 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883157    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883163    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883167    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883171    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883175    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883179    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883183    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883187    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883191    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883195    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883199    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883203    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883208    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883211    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883215    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883219    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883223    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883227    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883231    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883235    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:23.888023 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883248    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883252    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883256    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883261    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883265    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883271    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883276    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883281    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883286    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883291    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883296    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883300    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883305    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883309    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883314    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883318    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883322    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883326    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883329    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:23.888684 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883334    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883338    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883343    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883347    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883351    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883356    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883360    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883364    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883369    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883374    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883378    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883385    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883391    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883395    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883408    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883412    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883416    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:23.889238 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883420    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.883429    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883700    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883712    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883716    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883722    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883726    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883733    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883739    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883744    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883749    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883755    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883761    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883766    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:23.889737 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883780    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883785    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883789    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883793    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883798    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883802    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883806    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883811    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883815    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883819    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883823    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883828    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883832    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883837    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883841    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883845    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883857    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883862    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883866    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883870    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:23.890147 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883875    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883879    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883883    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883887    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883891    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883895    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883899    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883904    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883908    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883912    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883916    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883920    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883924    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883929    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883933    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883937    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883941    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883945    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883950    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:23.890828 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883954    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883958    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883962    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883967    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883971    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883975    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883982    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883987    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883991    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.883996    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884009    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884013    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884018    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884023    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884028    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884032    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884036    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884041    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884045    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:23.891661 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884049    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884053    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884058    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884062    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884066    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884070    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884073    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884077    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884098    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884102    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884106    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884110    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884114    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884118    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884122    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884126    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884131    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884136    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884140    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884144    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:23.892513 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884148    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884152    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884156    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884161    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884177    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884181    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884185    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884189    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:23.884194    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.884202    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.884424    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.888013    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.888810    2580 server.go:1019] "Starting client certificate rotation"
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.888919    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:40:23.893329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.888977    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:40:23.893707 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.891727    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:40:23.894517 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.894467    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:40:23.900616 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.900596    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:40:23.902302 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.902281    2580 log.go:25] "Validated CRI v1 image API"
Sep 17 17:40:23.903268 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.903250    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:40:23.908843 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.908819    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9effc06a-bf33-4462-a1ba-20e5b797009a:/dev/nvme0n1p3 f89b034d-3af3-4b84-b35b-167a23005fd3:/dev/nvme0n1p4]
Sep 17 17:40:23.908941 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.908841    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:40:23.914582 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.914450    2580 manager.go:216] Machine: {Timestamp:2026-09-17 17:40:23.913319821 +0000 UTC m=+0.252713409 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100033 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2b3cc84b77e913b803c31fd925f5 SystemUUID:ec2b2b3c-c84b-77e9-13b8-03c31fd925f5 BootID:df178f5c-1c9c-4fcc-b4ba-1c3ac9121147 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:ba:d7:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:ba:d7:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:99:ab:3c:23:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:40:23.914582 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.914580    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:40:23.914756 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.914742    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:40:23.915722 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.915691    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:40:23.915886 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.915724    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-39.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:40:23.916351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.916339    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:40:23.916351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.916352    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:40:23.916415 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.916359    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:40:23.916415 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.916374    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:40:23.917549 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.917536    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:40:23.919165 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.919143    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:40:23.921189 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921157    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:40:23.921509 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921498    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:40:23.921546 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921540    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:40:23.921578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921559    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:40:23.921578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921574    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:40:23.921663 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921587    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:40:23.921663 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921598    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:40:23.921996 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.921984    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:40:23.922029 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.922003    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:40:23.923765 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.923746    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:40:23.925789 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.925774    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:40:23.925865 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.925804    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:40:23.927370 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927356    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927376    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927382    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927387    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927395    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927401    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927414    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927420    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927426    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:40:23.927429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927431    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:40:23.927774 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927442    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:40:23.927774 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.927448    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:40:23.928777 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.928757    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:40:23.928866 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.928792    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:40:23.933238 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.933216    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-39.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:40:23.933351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.933295    2580 server.go:1268] "Started kubelet"
Sep 17 17:40:23.933411 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.933346    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:40:23.933488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.933414    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:40:23.933529 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.933486    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:40:23.933529 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.933514    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:40:23.933855 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.933828    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:40:23.934232 ip-10-0-139-39 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:40:23.934563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.934529    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:40:23.935560 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.935411    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:40:23.937553 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.937527    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:40:23.942880 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.942861    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:40:23.943280 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.943267    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:40:23.943427 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.942701    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-39.ec2.internal.18d62c7e8ad7e550  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-39.ec2.internal,UID:ip-10-0-139-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-39.ec2.internal,},FirstTimestamp:2026-09-17 17:40:23.93325704 +0000 UTC m=+0.272650622,LastTimestamp:2026-09-17 17:40:23.93325704 +0000 UTC m=+0.272650622,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-39.ec2.internal,}"
Sep 17 17:40:23.943560 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.943408    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:23.943876 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.943852    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:40:23.943960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.943856    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:40:23.943960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.943895    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:40:23.943960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.943951    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:40:23.944078 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.943964    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:40:23.944239 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944225    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:40:23.944279 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944242    2580 factory.go:55] Registering systemd factory
Sep 17 17:40:23.944279 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944252    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:40:23.944488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944468    2580 factory.go:153] Registering CRI-O factory
Sep 17 17:40:23.944488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944488    2580 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:40:23.944638 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944503    2580 factory.go:103] Registering Raw factory
Sep 17 17:40:23.944638 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944514    2580 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:40:23.944833 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.944822    2580 manager.go:349] Starting recovery of all containers
Sep 17 17:40:23.956386 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.956358    2580 manager.go:354] Recovery completed
Sep 17 17:40:23.958404 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.958378    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:40:23.958509 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.958405    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:40:23.958619 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.958599    2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 17:40:23.961891 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.961872    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:23.963861 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.963837    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:23.963940 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.963881    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:23.963940 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.963892    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:23.964546 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.964528    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:40:23.964546 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.964543    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:23.964639 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.964544    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:40:23.964639 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.964611    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:40:23.965225 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.965210    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:23.965276 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.965238    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:23.965276 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.965248    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:23.966866 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.966854    2580 policy_none.go:49] "None policy: Start"
Sep 17 17:40:23.966914 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.966870    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:40:23.966914 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.966881    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:40:23.968927 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.968912    2580 policy_none.go:47] "Start"
Sep 17 17:40:23.973557 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.973467    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-39.ec2.internal.18d62c7e8caadd51  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-39.ec2.internal,UID:ip-10-0-139-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-39.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-39.ec2.internal,},FirstTimestamp:2026-09-17 17:40:23.963860305 +0000 UTC m=+0.303253895,LastTimestamp:2026-09-17 17:40:23.963860305 +0000 UTC m=+0.303253895,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-39.ec2.internal,}"
Sep 17 17:40:23.975158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.975137    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xpv8l"
Sep 17 17:40:23.976137 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.976056    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-39.ec2.internal.18d62c7e8cab443a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-39.ec2.internal,UID:ip-10-0-139-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-39.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-39.ec2.internal,},FirstTimestamp:2026-09-17 17:40:23.96388665 +0000 UTC m=+0.303280235,LastTimestamp:2026-09-17 17:40:23.96388665 +0000 UTC m=+0.303280235,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-39.ec2.internal,}"
Sep 17 17:40:23.983359 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:23.983340    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xpv8l"
Sep 17 17:40:23.983555 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:23.983484    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-39.ec2.internal.18d62c7e8cab6ae1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-39.ec2.internal,UID:ip-10-0-139-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-39.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-39.ec2.internal,},FirstTimestamp:2026-09-17 17:40:23.963896545 +0000 UTC m=+0.303290127,LastTimestamp:2026-09-17 17:40:23.963896545 +0000 UTC m=+0.303290127,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-39.ec2.internal,}"
Sep 17 17:40:24.004758 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.004740    2580 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.004793    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.004810    2580 server.go:87] "Starting device plugin registration server"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.005182    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.005195    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.005301    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.005401    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.005410    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.005953    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:40:24.025224 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.006010    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.037872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.037829    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:40:24.039220 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.039201    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:40:24.039327 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.039263    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:40:24.039327 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.039297    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:40:24.039417 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.039348    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:40:24.041207 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.041181    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:24.106364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.106330    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:24.107396 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.107376    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:24.107536 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.107413    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:24.107536 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.107428    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:24.107536 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.107459    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.117178 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.117070    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.117178 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.117116    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-39.ec2.internal\": node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.140252 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.140215    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal"]
Sep 17 17:40:24.140431 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.140334    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:24.141910 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.141896    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:24.141979 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.141926    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:24.141979 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.141936    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:24.144423 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.144408    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:24.144489 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.144431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0e6c1c4cb2e164b7ccc9c7d72d5da826-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"0e6c1c4cb2e164b7ccc9c7d72d5da826\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.144489 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.144461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e6c1c4cb2e164b7ccc9c7d72d5da826-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"0e6c1c4cb2e164b7ccc9c7d72d5da826\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.144585 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.144558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.144653 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.144591    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:24.145307 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.145289    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:24.145422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.145320    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:24.145422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.145332    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:24.145422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.145288    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:24.145422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.145406    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:24.145422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.145420    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:24.147724 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.147709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.147808 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.147737    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:24.148538 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.148517    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:24.148639 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.148553    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:24.148639 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.148564    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:24.150437 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.150421    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.180219 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.180186    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-39.ec2.internal\" not found" node="ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.185305 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.185284    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-39.ec2.internal\" not found" node="ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.245451 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.245414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0e6c1c4cb2e164b7ccc9c7d72d5da826-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"0e6c1c4cb2e164b7ccc9c7d72d5da826\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.245451 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.245456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e6c1c4cb2e164b7ccc9c7d72d5da826-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"0e6c1c4cb2e164b7ccc9c7d72d5da826\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.245693 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.245481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46cd3fb458d38324dc66491843cdea97-config\") pod \"kube-apiserver-proxy-ip-10-0-139-39.ec2.internal\" (UID: \"46cd3fb458d38324dc66491843cdea97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.245693 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.245512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e6c1c4cb2e164b7ccc9c7d72d5da826-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"0e6c1c4cb2e164b7ccc9c7d72d5da826\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.245693 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.245515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0e6c1c4cb2e164b7ccc9c7d72d5da826-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"0e6c1c4cb2e164b7ccc9c7d72d5da826\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.250974 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.250940    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.346230 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.346186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46cd3fb458d38324dc66491843cdea97-config\") pod \"kube-apiserver-proxy-ip-10-0-139-39.ec2.internal\" (UID: \"46cd3fb458d38324dc66491843cdea97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.346422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.346252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46cd3fb458d38324dc66491843cdea97-config\") pod \"kube-apiserver-proxy-ip-10-0-139-39.ec2.internal\" (UID: \"46cd3fb458d38324dc66491843cdea97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.351338 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.351303    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.452310 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.452199    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.483387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.483361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.489120 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.488762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:24.553001 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.552943    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.653539 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.653482    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.754218 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.754114    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.854722 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.854687    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.888903 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.888866    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:40:24.889427 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.889062    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:40:24.954901 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:24.954876    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:24.988277 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.988224    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:35:23 +0000 UTC" deadline="2028-03-31 21:48:42.433179495 +0000 UTC"
Sep 17 17:40:24.988277 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:24.988275    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13468h8m17.44490962s"
Sep 17 17:40:25.003419 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.003389    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:25.004704 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.004655    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:25.004704 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.004687    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:25.004704 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.004697    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:25.055225 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.055190    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:25.082653 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:25.082470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6c1c4cb2e164b7ccc9c7d72d5da826.slice/crio-da9abf263c6d558ca513cac10516af50d8257db643911f95d270c9b44d9f6c42 WatchSource:0}: Error finding container da9abf263c6d558ca513cac10516af50d8257db643911f95d270c9b44d9f6c42: Status 404 returned error can't find the container with id da9abf263c6d558ca513cac10516af50d8257db643911f95d270c9b44d9f6c42
Sep 17 17:40:25.083126 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:25.083106    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cd3fb458d38324dc66491843cdea97.slice/crio-d43cf1639826870caf79b70b08fd5ab63b469d747b8b91f4581d0acd3aab2d76 WatchSource:0}: Error finding container d43cf1639826870caf79b70b08fd5ab63b469d747b8b91f4581d0acd3aab2d76: Status 404 returned error can't find the container with id d43cf1639826870caf79b70b08fd5ab63b469d747b8b91f4581d0acd3aab2d76
Sep 17 17:40:25.089231 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.088999    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:40:25.147368 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.147319    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:25.156040 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.155995    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:25.256639 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.256544    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:25.357171 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.357139    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:25.375648 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.375619    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:25.457391 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.457351    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found"
Sep 17 17:40:25.532960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.532874    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:25.544071 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.544047    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:25.557251 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.557227    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:40:25.557984 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.557970    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal"
Sep 17 17:40:25.566732 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.566700    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:40:25.916934 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.916794    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:25.922458 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.922416    2580 apiserver.go:52] "Watching apiserver"
Sep 17 17:40:25.932405 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.932363    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:40:25.933176 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.933133    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-tx7b2","openshift-multus/network-metrics-daemon-d67rh","kube-system/global-pull-secret-syncer-2tnhn","openshift-image-registry/node-ca-8hxlp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal","openshift-network-diagnostics/network-check-target-z689h","openshift-network-operator/iptables-alerter-4c7sm","openshift-ovn-kubernetes/ovnkube-node-4mftf","kube-system/konnectivity-agent-k6dnj","kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb","openshift-cluster-node-tuning-operator/tuned-gmdmc","openshift-dns/node-resolver-f2lj4","openshift-multus/multus-additional-cni-plugins-tz442"]
Sep 17 17:40:25.936169 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.936132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:25.939099 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.938981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.940014 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.939978    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:40:25.940014 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.939979    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z6cb9\""
Sep 17 17:40:25.940351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.939986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:40:25.940351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.940117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:40:25.941962 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.941410    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:40:25.941962 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.941591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:40:25.941962 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.941663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:25.941962 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.941750    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:25.942313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.942047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:40:25.942313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.942078    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b6jq5\""
Sep 17 17:40:25.944471 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.944437    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:40:25.945378 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.945338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:25.947261 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.947111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.951765 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.951722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:25.951960 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.951849    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.952506    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.952649    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.952754    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.953302    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.953386    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.953489    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cptkl\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.953772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.953833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.953957    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:40:25.955617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.954324    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-llgzl\""
Sep 17 17:40:25.956227 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.955987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtk4\" (UniqueName: \"kubernetes.io/projected/877c5428-fe8b-4287-beae-79dea1c8bd62-kube-api-access-9gtk4\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-socket-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-device-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f49d3fa-4582-4508-8858-4596395b5f57-host\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f49d3fa-4582-4508-8858-4596395b5f57-serviceca\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-system-cni-dir\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/877c5428-fe8b-4287-beae-79dea1c8bd62-host-slash\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-sys-fs\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45cd1091-7a05-4050-9ea5-fc2c920202bf-kubelet-config\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.956983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/877c5428-fe8b-4287-beae-79dea1c8bd62-iptables-alerter-script\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:25.957859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94z4k\" (UniqueName: \"kubernetes.io/projected/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-kube-api-access-94z4k\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.958750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45cd1091-7a05-4050-9ea5-fc2c920202bf-dbus\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:25.958750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-cnibin\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.958750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.958750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xt5\" (UniqueName: \"kubernetes.io/projected/4a333d12-515a-4448-a372-27444bf5cb3f-kube-api-access-c7xt5\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:25.958750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-registration-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:25.958750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:25.958750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.957458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j896\" (UniqueName: \"kubernetes.io/projected/3f49d3fa-4582-4508-8858-4596395b5f57-kube-api-access-8j896\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:25.959098 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.958791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:25.960369 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.960066    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:40:25.960369 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.960360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:40:25.960550 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.960453    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cbmx2\""
Sep 17 17:40:25.961405 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.961362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:40:25.961586 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.961571    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kb52x\""
Sep 17 17:40:25.961797 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.961779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:40:25.962112 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.962096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:25.962231 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:25.962207    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:25.964628 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.964606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:25.965392 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.965368    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:40:25.968387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.968364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:25.969268 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.969247    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:40:25.969516 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.969497    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:40:25.969516 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.969514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:40:25.969720 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.969694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:40:25.970067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.970050    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:40:25.970846 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.970826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ktglk\""
Sep 17 17:40:25.970984 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.970965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:25.971659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.971642    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:40:25.971992 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.971967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:40:25.972144 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.971976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jw6qb\""
Sep 17 17:40:25.973764 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.973659    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:40:25.974003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.973982    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:40:25.974072 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.974013    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-drqnh\""
Sep 17 17:40:25.986632 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.986588    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgp7l"
Sep 17 17:40:25.995165 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:25.995136    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgp7l"
Sep 17 17:40:26.045103 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.045061    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:40:26.046668 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.046606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" event={"ID":"0e6c1c4cb2e164b7ccc9c7d72d5da826","Type":"ContainerStarted","Data":"da9abf263c6d558ca513cac10516af50d8257db643911f95d270c9b44d9f6c42"}
Sep 17 17:40:26.047841 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.047806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" event={"ID":"46cd3fb458d38324dc66491843cdea97","Type":"ContainerStarted","Data":"d43cf1639826870caf79b70b08fd5ab63b469d747b8b91f4581d0acd3aab2d76"}
Sep 17 17:40:26.057728 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.057893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnqm\" (UniqueName: \"kubernetes.io/projected/78c77eba-b3f9-4472-8dad-5b68b500e7dd-kube-api-access-xnnqm\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:26.057893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovn-node-metrics-cert\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.057893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-cni-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.057893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/877c5428-fe8b-4287-beae-79dea1c8bd62-host-slash\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:26.057893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-etc-kubernetes\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.057893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysctl-d\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/877c5428-fe8b-4287-beae-79dea1c8bd62-host-slash\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.057938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94z4k\" (UniqueName: \"kubernetes.io/projected/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-kube-api-access-94z4k\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-system-cni-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysconfig\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-var-lib-kubelet\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-tmp\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fef4bc-4131-4d25-b641-686631df7b5e-tmp-dir\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.058166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-cni-bin\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j896\" (UniqueName: \"kubernetes.io/projected/3f49d3fa-4582-4508-8858-4596395b5f57-kube-api-access-8j896\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-systemd\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-node-log\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.058324    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-log-socket\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-modprobe-d\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.058447    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret podName:45cd1091-7a05-4050-9ea5-fc2c920202bf nodeName:}" failed. No retries permitted until 2026-09-17 17:40:26.558425578 +0000 UTC m=+2.897819171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret") pod "global-pull-secret-syncer-2tnhn" (UID: "45cd1091-7a05-4050-9ea5-fc2c920202bf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f49d3fa-4582-4508-8858-4596395b5f57-serviceca\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.058571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-slash\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-ovn\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-systemd\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40439aa5-ea91-4890-8db7-7504f4047dff-agent-certs\") pod \"konnectivity-agent-k6dnj\" (UID: \"40439aa5-ea91-4890-8db7-7504f4047dff\") " pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-sys-fs\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-sys-fs\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-run-netns\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-var-lib-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/877c5428-fe8b-4287-beae-79dea1c8bd62-iptables-alerter-script\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wkr\" (UniqueName: \"kubernetes.io/projected/79df6c0a-271d-41dd-aab9-02fbedb94de1-kube-api-access-z5wkr\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-cni-multus\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-kubelet\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.058996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f49d3fa-4582-4508-8858-4596395b5f57-serviceca\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-registration-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-etc-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-netns\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.059249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-registration-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-hostroot\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtk4\" (UniqueName: \"kubernetes.io/projected/877c5428-fe8b-4287-beae-79dea1c8bd62-kube-api-access-9gtk4\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-device-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-run-ovn-kubernetes\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-cni-netd\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-device-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-env-overrides\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-multus-certs\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-run\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovnkube-script-lib\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-daemon-config\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-lib-modules\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.060123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-cnibin\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/877c5428-fe8b-4287-beae-79dea1c8bd62-iptables-alerter-script\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-kubelet\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-cnibin\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-cni-bin\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-cnibin\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysctl-conf\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-sys\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-systemd-units\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40439aa5-ea91-4890-8db7-7504f4047dff-konnectivity-ca\") pod \"konnectivity-agent-k6dnj\" (UID: \"40439aa5-ea91-4890-8db7-7504f4047dff\") " pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23fef4bc-4131-4d25-b641-686631df7b5e-hosts-file\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4qq\" (UniqueName: \"kubernetes.io/projected/23fef4bc-4131-4d25-b641-686631df7b5e-kube-api-access-cj4qq\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.060804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-socket-dir-parent\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45cd1091-7a05-4050-9ea5-fc2c920202bf-kubelet-config\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovnkube-config\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45cd1091-7a05-4050-9ea5-fc2c920202bf-dbus\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xt5\" (UniqueName: \"kubernetes.io/projected/4a333d12-515a-4448-a372-27444bf5cb3f-kube-api-access-c7xt5\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddfa57a6-109d-430c-b75d-cccf0b616d95-cni-binary-copy\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-conf-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7g8n\" (UniqueName: \"kubernetes.io/projected/ddfa57a6-109d-430c-b75d-cccf0b616d95-kube-api-access-w7g8n\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-kubernetes\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45cd1091-7a05-4050-9ea5-fc2c920202bf-dbus\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-host\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h528f\" (UniqueName: \"kubernetes.io/projected/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-kube-api-access-h528f\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.059910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45cd1091-7a05-4050-9ea5-fc2c920202bf-kubelet-config\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-tuned\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-k8s-cni-cncf-io\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-socket-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.061365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a333d12-515a-4448-a372-27444bf5cb3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.061987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f49d3fa-4582-4508-8858-4596395b5f57-host\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:26.061987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-system-cni-dir\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.061987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f49d3fa-4582-4508-8858-4596395b5f57-host\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:26.061987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a333d12-515a-4448-a372-27444bf5cb3f-system-cni-dir\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.061987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.060417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-socket-dir\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.067336 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.067310    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:40:26.072566 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.072519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j896\" (UniqueName: \"kubernetes.io/projected/3f49d3fa-4582-4508-8858-4596395b5f57-kube-api-access-8j896\") pod \"node-ca-8hxlp\" (UID: \"3f49d3fa-4582-4508-8858-4596395b5f57\") " pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:26.072961 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.072912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94z4k\" (UniqueName: \"kubernetes.io/projected/a4322432-514d-4d7c-a5ed-4ad2d5552e2f-kube-api-access-94z4k\") pod \"aws-ebs-csi-driver-node-lv4zb\" (UID: \"a4322432-514d-4d7c-a5ed-4ad2d5552e2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.075066 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.075038    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtk4\" (UniqueName: \"kubernetes.io/projected/877c5428-fe8b-4287-beae-79dea1c8bd62-kube-api-access-9gtk4\") pod \"iptables-alerter-4c7sm\" (UID: \"877c5428-fe8b-4287-beae-79dea1c8bd62\") " pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:26.075462 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.075438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xt5\" (UniqueName: \"kubernetes.io/projected/4a333d12-515a-4448-a372-27444bf5cb3f-kube-api-access-c7xt5\") pod \"multus-additional-cni-plugins-tz442\" (UID: \"4a333d12-515a-4448-a372-27444bf5cb3f\") " pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.160810 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.160764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovnkube-config\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.160836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddfa57a6-109d-430c-b75d-cccf0b616d95-cni-binary-copy\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.160865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-conf-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.160896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7g8n\" (UniqueName: \"kubernetes.io/projected/ddfa57a6-109d-430c-b75d-cccf0b616d95-kube-api-access-w7g8n\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.160965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-kubernetes\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.160985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-conf-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-host\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h528f\" (UniqueName: \"kubernetes.io/projected/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-kube-api-access-h528f\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-tuned\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-k8s-cni-cncf-io\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnqm\" (UniqueName: \"kubernetes.io/projected/78c77eba-b3f9-4472-8dad-5b68b500e7dd-kube-api-access-xnnqm\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovn-node-metrics-cert\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-kubernetes\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-cni-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-etc-kubernetes\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysctl-d\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-system-cni-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161366 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysconfig\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-var-lib-kubelet\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-tmp\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fef4bc-4131-4d25-b641-686631df7b5e-tmp-dir\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-cni-bin\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-host\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovnkube-config\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-systemd\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-cni-bin\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysconfig\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-node-log\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-log-socket\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysctl-d\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-modprobe-d\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-var-lib-kubelet\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-slash\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-slash\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddfa57a6-109d-430c-b75d-cccf0b616d95-cni-binary-copy\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.161895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-node-log\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-log-socket\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.161951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-modprobe-d\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-etc-kubernetes\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-k8s-cni-cncf-io\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-system-cni-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-systemd\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fef4bc-4131-4d25-b641-686631df7b5e-tmp-dir\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-cni-dir\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-ovn\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-systemd\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-ovn\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40439aa5-ea91-4890-8db7-7504f4047dff-agent-certs\") pod \"konnectivity-agent-k6dnj\" (UID: \"40439aa5-ea91-4890-8db7-7504f4047dff\") " pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-systemd\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.162806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-run-netns\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-var-lib-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-run-netns\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wkr\" (UniqueName: \"kubernetes.io/projected/79df6c0a-271d-41dd-aab9-02fbedb94de1-kube-api-access-z5wkr\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-cni-multus\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-kubelet\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-etc-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-netns\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-run-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-cni-multus\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-hostroot\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-etc-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-netns\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-var-lib-openvswitch\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.162990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-hostroot\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-run-ovn-kubernetes\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.163469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-cni-netd\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-env-overrides\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-var-lib-kubelet\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-multus-certs\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-run\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-run-ovn-kubernetes\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovnkube-script-lib\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-daemon-config\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-lib-modules\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-kubelet\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.163430    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-host-run-multus-certs\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.163507    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:40:26.663488054 +0000 UTC m=+3.002881639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-run\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-cni-bin\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-cnibin\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.164295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysctl-conf\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-sys\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-systemd-units\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40439aa5-ea91-4890-8db7-7504f4047dff-konnectivity-ca\") pod \"konnectivity-agent-k6dnj\" (UID: \"40439aa5-ea91-4890-8db7-7504f4047dff\") " pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23fef4bc-4131-4d25-b641-686631df7b5e-hosts-file\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4qq\" (UniqueName: \"kubernetes.io/projected/23fef4bc-4131-4d25-b641-686631df7b5e-kube-api-access-cj4qq\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-socket-dir-parent\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-env-overrides\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.163934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23fef4bc-4131-4d25-b641-686631df7b5e-hosts-file\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-sys\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-cni-bin\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-systemd-units\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-sysctl-conf\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-etc-tuned\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-cnibin\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-lib-modules\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40439aa5-ea91-4890-8db7-7504f4047dff-konnectivity-ca\") pod \"konnectivity-agent-k6dnj\" (UID: \"40439aa5-ea91-4890-8db7-7504f4047dff\") " pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:26.165027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovn-node-metrics-cert\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-cni-netd\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-socket-dir-parent\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.165659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79df6c0a-271d-41dd-aab9-02fbedb94de1-host-kubelet\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.164853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddfa57a6-109d-430c-b75d-cccf0b616d95-multus-daemon-config\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.165659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.165064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-tmp\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.165659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.165183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79df6c0a-271d-41dd-aab9-02fbedb94de1-ovnkube-script-lib\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.165924 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.165722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40439aa5-ea91-4890-8db7-7504f4047dff-agent-certs\") pod \"konnectivity-agent-k6dnj\" (UID: \"40439aa5-ea91-4890-8db7-7504f4047dff\") " pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:26.168320 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.168246    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:26.168320 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.168279    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:26.168320 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.168294    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dh9v6 for pod openshift-network-diagnostics/network-check-target-z689h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:26.168544 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.168361    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6 podName:df61c5cc-b897-402c-9cfd-1c7b02a8419f nodeName:}" failed. No retries permitted until 2026-09-17 17:40:26.668344445 +0000 UTC m=+3.007738028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dh9v6" (UniqueName: "kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6") pod "network-check-target-z689h" (UID: "df61c5cc-b897-402c-9cfd-1c7b02a8419f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:26.170876 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.170805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7g8n\" (UniqueName: \"kubernetes.io/projected/ddfa57a6-109d-430c-b75d-cccf0b616d95-kube-api-access-w7g8n\") pod \"multus-tx7b2\" (UID: \"ddfa57a6-109d-430c-b75d-cccf0b616d95\") " pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.170876 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.170829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h528f\" (UniqueName: \"kubernetes.io/projected/a4619b3c-aef9-4cc0-ae72-de0f8b27fcde-kube-api-access-h528f\") pod \"tuned-gmdmc\" (UID: \"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde\") " pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.171978 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.171955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wkr\" (UniqueName: \"kubernetes.io/projected/79df6c0a-271d-41dd-aab9-02fbedb94de1-kube-api-access-z5wkr\") pod \"ovnkube-node-4mftf\" (UID: \"79df6c0a-271d-41dd-aab9-02fbedb94de1\") " pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.172461 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.172443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4qq\" (UniqueName: \"kubernetes.io/projected/23fef4bc-4131-4d25-b641-686631df7b5e-kube-api-access-cj4qq\") pod \"node-resolver-f2lj4\" (UID: \"23fef4bc-4131-4d25-b641-686631df7b5e\") " pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.173362 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.173338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnqm\" (UniqueName: \"kubernetes.io/projected/78c77eba-b3f9-4472-8dad-5b68b500e7dd-kube-api-access-xnnqm\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:26.264757 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.264704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4c7sm"
Sep 17 17:40:26.273623 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.273595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb"
Sep 17 17:40:26.284499 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.284467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hxlp"
Sep 17 17:40:26.292456 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.292428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tz442"
Sep 17 17:40:26.302410 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.302376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:26.310363 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.310339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f2lj4"
Sep 17 17:40:26.317348 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.317322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:26.325190 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.325166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tx7b2"
Sep 17 17:40:26.330996 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.330971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gmdmc"
Sep 17 17:40:26.567159 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.567045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:26.567330 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.567246    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:26.567330 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.567323    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret podName:45cd1091-7a05-4050-9ea5-fc2c920202bf nodeName:}" failed. No retries permitted until 2026-09-17 17:40:27.567305478 +0000 UTC m=+3.906699067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret") pod "global-pull-secret-syncer-2tnhn" (UID: "45cd1091-7a05-4050-9ea5-fc2c920202bf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:26.668147 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.668109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:26.668372 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.668309    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:26.668446 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.668405    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:40:27.668388517 +0000 UTC m=+4.007782093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:26.769401 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.769366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:26.769583 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.769560    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:26.769629 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.769590    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:26.769629 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.769602    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dh9v6 for pod openshift-network-diagnostics/network-check-target-z689h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:26.769700 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:26.769668    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6 podName:df61c5cc-b897-402c-9cfd-1c7b02a8419f nodeName:}" failed. No retries permitted until 2026-09-17 17:40:27.769653303 +0000 UTC m=+4.109046886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh9v6" (UniqueName: "kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6") pod "network-check-target-z689h" (UID: "df61c5cc-b897-402c-9cfd-1c7b02a8419f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:26.998862 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.998792    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:35:25 +0000 UTC" deadline="2028-03-29 23:49:13.04352654 +0000 UTC"
Sep 17 17:40:26.998862 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:26.998857    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13422h8m46.044673892s"
Sep 17 17:40:27.040282 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:27.040239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:27.040491 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.040378    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:27.141073 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.141035    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4619b3c_aef9_4cc0_ae72_de0f8b27fcde.slice/crio-db8f64007049ee3453ef272ebea1e2f4d4c111dc6088ab9675e6b0d072569b78 WatchSource:0}: Error finding container db8f64007049ee3453ef272ebea1e2f4d4c111dc6088ab9675e6b0d072569b78: Status 404 returned error can't find the container with id db8f64007049ee3453ef272ebea1e2f4d4c111dc6088ab9675e6b0d072569b78
Sep 17 17:40:27.142164 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.142139    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f49d3fa_4582_4508_8858_4596395b5f57.slice/crio-f517c74f098849119d471809643983a7044806d1a5ffbed2b06b4c3cc9f5637b WatchSource:0}: Error finding container f517c74f098849119d471809643983a7044806d1a5ffbed2b06b4c3cc9f5637b: Status 404 returned error can't find the container with id f517c74f098849119d471809643983a7044806d1a5ffbed2b06b4c3cc9f5637b
Sep 17 17:40:27.143924 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.143829    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fef4bc_4131_4d25_b641_686631df7b5e.slice/crio-c92a88d186ba453dc9e672f14445fbe572660a77fa49462a88679c470cafd13d WatchSource:0}: Error finding container c92a88d186ba453dc9e672f14445fbe572660a77fa49462a88679c470cafd13d: Status 404 returned error can't find the container with id c92a88d186ba453dc9e672f14445fbe572660a77fa49462a88679c470cafd13d
Sep 17 17:40:27.144845 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.144778    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79df6c0a_271d_41dd_aab9_02fbedb94de1.slice/crio-714ba63a146d989eed95c28a112ae0f924867147ccc79147614eeae1807d8a1a WatchSource:0}: Error finding container 714ba63a146d989eed95c28a112ae0f924867147ccc79147614eeae1807d8a1a: Status 404 returned error can't find the container with id 714ba63a146d989eed95c28a112ae0f924867147ccc79147614eeae1807d8a1a
Sep 17 17:40:27.149011 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.148680    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfa57a6_109d_430c_b75d_cccf0b616d95.slice/crio-b7278f6ad0ecbf22f3220d5a15ea68dec7880c22e781cc8ca538985393cd2b14 WatchSource:0}: Error finding container b7278f6ad0ecbf22f3220d5a15ea68dec7880c22e781cc8ca538985393cd2b14: Status 404 returned error can't find the container with id b7278f6ad0ecbf22f3220d5a15ea68dec7880c22e781cc8ca538985393cd2b14
Sep 17 17:40:27.149773 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.149693    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877c5428_fe8b_4287_beae_79dea1c8bd62.slice/crio-1a3cc533847a29936bd77b21136fe24e3fffb66df57590ab2e7a19f533255a6e WatchSource:0}: Error finding container 1a3cc533847a29936bd77b21136fe24e3fffb66df57590ab2e7a19f533255a6e: Status 404 returned error can't find the container with id 1a3cc533847a29936bd77b21136fe24e3fffb66df57590ab2e7a19f533255a6e
Sep 17 17:40:27.150756 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.150730    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4322432_514d_4d7c_a5ed_4ad2d5552e2f.slice/crio-c87f8ac95d20338cad1924e02914f1beb7c2c8634d89ffd12fa074f8cce2faa8 WatchSource:0}: Error finding container c87f8ac95d20338cad1924e02914f1beb7c2c8634d89ffd12fa074f8cce2faa8: Status 404 returned error can't find the container with id c87f8ac95d20338cad1924e02914f1beb7c2c8634d89ffd12fa074f8cce2faa8
Sep 17 17:40:27.151742 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.151710    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40439aa5_ea91_4890_8db7_7504f4047dff.slice/crio-0f0c18d2786d0a2b8d17d001c02cc1b4ed6ea1175bc533b4235d6c52b13d2217 WatchSource:0}: Error finding container 0f0c18d2786d0a2b8d17d001c02cc1b4ed6ea1175bc533b4235d6c52b13d2217: Status 404 returned error can't find the container with id 0f0c18d2786d0a2b8d17d001c02cc1b4ed6ea1175bc533b4235d6c52b13d2217
Sep 17 17:40:27.152886 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:27.152667    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a333d12_515a_4448_a372_27444bf5cb3f.slice/crio-39123696d9f8a700d8cff4eeec8242bef430e0e77565a0445eb497123a181da1 WatchSource:0}: Error finding container 39123696d9f8a700d8cff4eeec8242bef430e0e77565a0445eb497123a181da1: Status 404 returned error can't find the container with id 39123696d9f8a700d8cff4eeec8242bef430e0e77565a0445eb497123a181da1
Sep 17 17:40:27.576428 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:27.576324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:27.576616 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.576478    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:27.576616 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.576565    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret podName:45cd1091-7a05-4050-9ea5-fc2c920202bf nodeName:}" failed. No retries permitted until 2026-09-17 17:40:29.576543603 +0000 UTC m=+5.915937188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret") pod "global-pull-secret-syncer-2tnhn" (UID: "45cd1091-7a05-4050-9ea5-fc2c920202bf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:27.677895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:27.677134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:27.677895 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.677381    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:27.677895 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.677459    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:40:29.677435195 +0000 UTC m=+6.016828786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:27.777893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:27.777824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:27.778176 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.778038    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:27.778176 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.778061    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:27.778176 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.778074    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dh9v6 for pod openshift-network-diagnostics/network-check-target-z689h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:27.778176 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:27.778159    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6 podName:df61c5cc-b897-402c-9cfd-1c7b02a8419f nodeName:}" failed. No retries permitted until 2026-09-17 17:40:29.778138652 +0000 UTC m=+6.117532238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh9v6" (UniqueName: "kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6") pod "network-check-target-z689h" (UID: "df61c5cc-b897-402c-9cfd-1c7b02a8419f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:27.999737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:27.999623    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:35:25 +0000 UTC" deadline="2028-03-01 02:06:27.844196968 +0000 UTC"
Sep 17 17:40:27.999737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:27.999676    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12728h25m59.844526905s"
Sep 17 17:40:28.042543 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.042503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:28.042722 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:28.042658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:28.043109 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.043074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:28.043207 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:28.043187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:28.058543 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.058427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tx7b2" event={"ID":"ddfa57a6-109d-430c-b75d-cccf0b616d95","Type":"ContainerStarted","Data":"b7278f6ad0ecbf22f3220d5a15ea68dec7880c22e781cc8ca538985393cd2b14"}
Sep 17 17:40:28.066369 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.066249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f2lj4" event={"ID":"23fef4bc-4131-4d25-b641-686631df7b5e","Type":"ContainerStarted","Data":"c92a88d186ba453dc9e672f14445fbe572660a77fa49462a88679c470cafd13d"}
Sep 17 17:40:28.089015 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.088973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hxlp" event={"ID":"3f49d3fa-4582-4508-8858-4596395b5f57","Type":"ContainerStarted","Data":"f517c74f098849119d471809643983a7044806d1a5ffbed2b06b4c3cc9f5637b"}
Sep 17 17:40:28.097563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.097518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6dnj" event={"ID":"40439aa5-ea91-4890-8db7-7504f4047dff","Type":"ContainerStarted","Data":"0f0c18d2786d0a2b8d17d001c02cc1b4ed6ea1175bc533b4235d6c52b13d2217"}
Sep 17 17:40:28.111307 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.111258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb" event={"ID":"a4322432-514d-4d7c-a5ed-4ad2d5552e2f","Type":"ContainerStarted","Data":"c87f8ac95d20338cad1924e02914f1beb7c2c8634d89ffd12fa074f8cce2faa8"}
Sep 17 17:40:28.117058 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.117015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"714ba63a146d989eed95c28a112ae0f924867147ccc79147614eeae1807d8a1a"}
Sep 17 17:40:28.125443 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.125398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gmdmc" event={"ID":"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde","Type":"ContainerStarted","Data":"db8f64007049ee3453ef272ebea1e2f4d4c111dc6088ab9675e6b0d072569b78"}
Sep 17 17:40:28.141408 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.141365    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e6c1c4cb2e164b7ccc9c7d72d5da826" containerID="edbabec536e61474a7ff0b058cea50b6c9f81f28b71dd2c185d8885bf55ab30a" exitCode=0
Sep 17 17:40:28.141598 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.141478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" event={"ID":"0e6c1c4cb2e164b7ccc9c7d72d5da826","Type":"ContainerDied","Data":"edbabec536e61474a7ff0b058cea50b6c9f81f28b71dd2c185d8885bf55ab30a"}
Sep 17 17:40:28.161441 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.161399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" event={"ID":"46cd3fb458d38324dc66491843cdea97","Type":"ContainerStarted","Data":"1134899423aa505bf13389def236e907ff4dfcf3dff811d3219240ede068202d"}
Sep 17 17:40:28.173584 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.173335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerStarted","Data":"39123696d9f8a700d8cff4eeec8242bef430e0e77565a0445eb497123a181da1"}
Sep 17 17:40:28.177172 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.177112    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" podStartSLOduration=3.17707327 podStartE2EDuration="3.17707327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:28.176653924 +0000 UTC m=+4.516047518" watchObservedRunningTime="2026-09-17 17:40:28.17707327 +0000 UTC m=+4.516466862"
Sep 17 17:40:28.183103 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:28.183055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4c7sm" event={"ID":"877c5428-fe8b-4287-beae-79dea1c8bd62","Type":"ContainerStarted","Data":"1a3cc533847a29936bd77b21136fe24e3fffb66df57590ab2e7a19f533255a6e"}
Sep 17 17:40:29.040832 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:29.040530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:29.041379 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.040963    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:29.216472 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:29.216442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" event={"ID":"0e6c1c4cb2e164b7ccc9c7d72d5da826","Type":"ContainerStarted","Data":"18f3300e785fe5f35f033d8dc757d43fb64ad1f14af6bc2734a466a0ea9b46df"}
Sep 17 17:40:29.595027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:29.594981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:29.595235 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.595165    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:29.595235 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.595234    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret podName:45cd1091-7a05-4050-9ea5-fc2c920202bf nodeName:}" failed. No retries permitted until 2026-09-17 17:40:33.595214839 +0000 UTC m=+9.934608431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret") pod "global-pull-secret-syncer-2tnhn" (UID: "45cd1091-7a05-4050-9ea5-fc2c920202bf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:29.695893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:29.695848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:29.696149 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.696024    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:29.696149 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.696120    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:40:33.69607741 +0000 UTC m=+10.035470996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:29.796686 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:29.796639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:29.797004 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.796979    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:29.797079 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.797013    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:29.797079 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.797029    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dh9v6 for pod openshift-network-diagnostics/network-check-target-z689h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:29.797219 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:29.797114    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6 podName:df61c5cc-b897-402c-9cfd-1c7b02a8419f nodeName:}" failed. No retries permitted until 2026-09-17 17:40:33.797074707 +0000 UTC m=+10.136468279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh9v6" (UniqueName: "kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6") pod "network-check-target-z689h" (UID: "df61c5cc-b897-402c-9cfd-1c7b02a8419f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:30.040574 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:30.040449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:30.040764 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:30.040627    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:30.041059 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:30.041036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:30.041533 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:30.041150    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:31.040660 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:31.040621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:31.040886 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:31.040792    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:32.040036 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:32.039994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:32.040538 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:32.040163    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:32.040599 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:32.040587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:32.040712 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:32.040688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:33.040072 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:33.040033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:33.040507 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.040206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:33.632867 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:33.632828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:33.633073 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.633000    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:33.633073 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.633071    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret podName:45cd1091-7a05-4050-9ea5-fc2c920202bf nodeName:}" failed. No retries permitted until 2026-09-17 17:40:41.633052175 +0000 UTC m=+17.972445745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret") pod "global-pull-secret-syncer-2tnhn" (UID: "45cd1091-7a05-4050-9ea5-fc2c920202bf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:33.733658 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:33.733616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:33.733866 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.733823    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:33.733932 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.733911    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:40:41.733889574 +0000 UTC m=+18.073283156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:33.834505 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:33.834460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:33.834729 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.834679    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:33.834729 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.834709    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:33.834729 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.834724    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dh9v6 for pod openshift-network-diagnostics/network-check-target-z689h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:33.834878 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:33.834797    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6 podName:df61c5cc-b897-402c-9cfd-1c7b02a8419f nodeName:}" failed. No retries permitted until 2026-09-17 17:40:41.834776043 +0000 UTC m=+18.174169632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh9v6" (UniqueName: "kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6") pod "network-check-target-z689h" (UID: "df61c5cc-b897-402c-9cfd-1c7b02a8419f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:34.041173 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:34.040959    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:34.041173 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:34.041102    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:34.041173 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:34.041149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:34.041747 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:34.041257    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:35.040536 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:35.040486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:35.040740 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:35.040657    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:36.045316 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:36.044599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:36.045316 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:36.044738    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:36.045316 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:36.045165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:36.045316 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:36.045268    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:37.039744 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:37.039705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:37.039927 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:37.039825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:38.040421 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:38.040361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:38.040902 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:38.040376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:38.040902 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:38.040510    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:38.040902 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:38.040585    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:39.039684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:39.039649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:39.039884 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:39.039792    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:40.039898 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:40.039846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:40.039898 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:40.039876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:40.040450 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:40.040011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:40.040450 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:40.040133    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:41.039727 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:41.039675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:41.040033 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.039831    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:41.700249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:41.700208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:41.700531 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.700357    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:41.700531 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.700442    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret podName:45cd1091-7a05-4050-9ea5-fc2c920202bf nodeName:}" failed. No retries permitted until 2026-09-17 17:40:57.700420215 +0000 UTC m=+34.039813849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret") pod "global-pull-secret-syncer-2tnhn" (UID: "45cd1091-7a05-4050-9ea5-fc2c920202bf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:41.801266 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:41.801228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:41.801456 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.801352    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:41.801456 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.801410    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:40:57.801394403 +0000 UTC m=+34.140787971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:41.901761 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:41.901718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:41.901966 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.901930    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:41.901966 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.901956    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:41.902065 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.901972    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dh9v6 for pod openshift-network-diagnostics/network-check-target-z689h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:41.902065 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:41.902042    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6 podName:df61c5cc-b897-402c-9cfd-1c7b02a8419f nodeName:}" failed. No retries permitted until 2026-09-17 17:40:57.902022722 +0000 UTC m=+34.241416362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh9v6" (UniqueName: "kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6") pod "network-check-target-z689h" (UID: "df61c5cc-b897-402c-9cfd-1c7b02a8419f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:42.040665 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:42.040566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:42.040665 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:42.040617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:42.041191 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:42.040736    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:42.041191 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:42.040861    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:43.039546 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:43.039503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:43.039765 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:43.039643    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:44.040966 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:44.040915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:44.041477 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:44.040980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:44.041477 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:44.041059    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:44.041477 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:44.041103    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:45.040148 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.040062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:45.040292 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:45.040245    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:45.252019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.251686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hxlp" event={"ID":"3f49d3fa-4582-4508-8858-4596395b5f57","Type":"ContainerStarted","Data":"5e2ca6660ade0f18a806083326c994d443df51e5ad5bbb179d06971140499667"}
Sep 17 17:40:45.253573 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.253540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6dnj" event={"ID":"40439aa5-ea91-4890-8db7-7504f4047dff","Type":"ContainerStarted","Data":"915aa3e7bf4c8bd08be0cd92b960ed83517a792d69796cd65f21dd45137e8243"}
Sep 17 17:40:45.255926 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.255793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb" event={"ID":"a4322432-514d-4d7c-a5ed-4ad2d5552e2f","Type":"ContainerStarted","Data":"f51e4d199b51b80d84eff208dc2b2eec6d4546e852df1547d92c34c7729d8f6a"}
Sep 17 17:40:45.257935 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.257895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"5cb599f89fc19ac2c31b673e1997843b69f0dd8fb894bfad86dc48973da43053"}
Sep 17 17:40:45.258061 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.257962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"eba10a423274352f588b10c966116979560fa1e39807f6f056db2d8ba4045dc9"}
Sep 17 17:40:45.260592 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.260553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gmdmc" event={"ID":"a4619b3c-aef9-4cc0-ae72-de0f8b27fcde","Type":"ContainerStarted","Data":"98d05005cd0106c37c403082122727260907c063ab5fe92f826f721feb08cce3"}
Sep 17 17:40:45.262256 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.262230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerStarted","Data":"b065e13a677a908bbe83f0e34cbebb7dab206cbdfa633cb1c356e0633fe4f6c6"}
Sep 17 17:40:45.263959 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.263929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tx7b2" event={"ID":"ddfa57a6-109d-430c-b75d-cccf0b616d95","Type":"ContainerStarted","Data":"c2fe06b5d4effbc0a550a0e13b160b978af717c003e0e34651160326eaf2bd5b"}
Sep 17 17:40:45.265551 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.265526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f2lj4" event={"ID":"23fef4bc-4131-4d25-b641-686631df7b5e","Type":"ContainerStarted","Data":"11175161a28f3e7b41233d4601dba90ed2dd95772069ad60a83a06a3560b1552"}
Sep 17 17:40:45.269580 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.269535    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8hxlp" podStartSLOduration=3.453354864 podStartE2EDuration="21.269521944s" totalImagesPullingTime="17.81616708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:45.269379213 +0000 UTC m=+21.608772801" watchObservedRunningTime="2026-09-17 17:40:45.269521944 +0000 UTC m=+21.608915534"
Sep 17 17:40:45.269676 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.269614    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" podStartSLOduration=20.269608084 podStartE2EDuration="20.269608084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:29.233796794 +0000 UTC m=+5.573190386" watchObservedRunningTime="2026-09-17 17:40:45.269608084 +0000 UTC m=+21.609001675"
Sep 17 17:40:45.289939 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.289879    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tx7b2" podStartSLOduration=3.415014166 podStartE2EDuration="21.289860191s" totalImagesPullingTime="17.874846025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:45.289570878 +0000 UTC m=+21.628964468" watchObservedRunningTime="2026-09-17 17:40:45.289860191 +0000 UTC m=+21.629253782"
Sep 17 17:40:45.312473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.312417    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gmdmc" podStartSLOduration=3.470195135 podStartE2EDuration="21.312399559s" totalImagesPullingTime="17.842204424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:45.312175099 +0000 UTC m=+21.651568690" watchObservedRunningTime="2026-09-17 17:40:45.312399559 +0000 UTC m=+21.651793150"
Sep 17 17:40:45.391216 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:45.391164    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k6dnj" podStartSLOduration=10.998252859 podStartE2EDuration="21.391144042s" totalImagesPullingTime="10.392891183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:45.360359449 +0000 UTC m=+21.699753060" watchObservedRunningTime="2026-09-17 17:40:45.391144042 +0000 UTC m=+21.730537653"
Sep 17 17:40:46.039897 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.039856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:46.040071 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.039857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:46.040071 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:46.040001    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:46.040071 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:46.040029    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:46.270060 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.270020    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a333d12-515a-4448-a372-27444bf5cb3f" containerID="b065e13a677a908bbe83f0e34cbebb7dab206cbdfa633cb1c356e0633fe4f6c6" exitCode=0
Sep 17 17:40:46.270508 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.270124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerDied","Data":"b065e13a677a908bbe83f0e34cbebb7dab206cbdfa633cb1c356e0633fe4f6c6"}
Sep 17 17:40:46.272275 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.272239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4c7sm" event={"ID":"877c5428-fe8b-4287-beae-79dea1c8bd62","Type":"ContainerStarted","Data":"3580a3ebb02a2e330d3d02b5646f5e51d98eddf97300ac256c7f5ea2239f990f"}
Sep 17 17:40:46.278049 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.278006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"320ebca00995ac20180562ba94fe452a997c52b7881b3cfaa892067f3a29fc6f"}
Sep 17 17:40:46.278215 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.278115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"4fa8a6637a050c2cb02a522a327adabdefd69ff683c715d6405215ec2da2b05d"}
Sep 17 17:40:46.278215 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.278150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"5028a78ab2739f92fadc15004f50a3edcf92e616ad40788cd4e78914e8c8eba7"}
Sep 17 17:40:46.278215 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.278185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"1863169db952b648422967927c2c552bd7a12da34561f0f74b4e29e3c5d06b1d"}
Sep 17 17:40:46.290396 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:46.290332    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f2lj4" podStartSLOduration=4.481896755 podStartE2EDuration="22.290307639s" totalImagesPullingTime="17.808410884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:45.390714729 +0000 UTC m=+21.730108319" watchObservedRunningTime="2026-09-17 17:40:46.290307639 +0000 UTC m=+22.629701231"
Sep 17 17:40:47.005596 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:47.005400    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:40:47.018221 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:47.018130    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:40:47.00558872Z","UUID":"5bdd35e5-cf1f-4f1f-b783-5116dd4eba5a","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:40:47.019799 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:47.019781    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:40:47.019904 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:47.019815    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:40:47.039935 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:47.039863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:47.040111 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:47.040018    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:47.282367 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:47.282250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb" event={"ID":"a4322432-514d-4d7c-a5ed-4ad2d5552e2f","Type":"ContainerStarted","Data":"f7afff1c0b9e7fdef210e2708dd0c0be02cc51f993163e7efe30a7297bfdd23c"}
Sep 17 17:40:47.435222 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:47.435177    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:48.040642 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:48.040558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:48.040809 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:48.040558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:48.040809 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:48.040697    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:48.040809 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:48.040743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:48.288351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:48.288306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"1bbcf49fa784c34650ea53f50dfd496251a4069efafd6ef3466de7ce0a56fd03"}
Sep 17 17:40:49.040317 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:49.040281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:49.040510 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:49.040429    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:49.997794 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:49.997740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:49.998653 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:49.998626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:50.016710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.016651    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4c7sm" podStartSLOduration=8.206758498 podStartE2EDuration="26.016633171s" totalImagesPullingTime="17.809874673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:46.290245652 +0000 UTC m=+22.629639243" watchObservedRunningTime="2026-09-17 17:40:50.016633171 +0000 UTC m=+26.356026763"
Sep 17 17:40:50.040501 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.040467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:50.040700 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:50.040599    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:50.040700 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.040641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:50.040795 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:50.040765    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:50.296877 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.296615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb" event={"ID":"a4322432-514d-4d7c-a5ed-4ad2d5552e2f","Type":"ContainerStarted","Data":"d4c39900827ded829e033bf62ceccb3f35f94fb1fcf95434cd32e8ea21b02f4a"}
Sep 17 17:40:50.301356 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.301242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" event={"ID":"79df6c0a-271d-41dd-aab9-02fbedb94de1","Type":"ContainerStarted","Data":"d62ca13db685759eb53d0b7f1f9e3df1dd5adc9f7a2cb8806d7a864342833fd4"}
Sep 17 17:40:50.302135 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.301982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k6dnj"
Sep 17 17:40:50.332590 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.332536    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lv4zb" podStartSLOduration=3.731563502 podStartE2EDuration="26.33251608s" totalImagesPullingTime="22.600952578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:50.315808245 +0000 UTC m=+26.655201847" watchObservedRunningTime="2026-09-17 17:40:50.33251608 +0000 UTC m=+26.671909670"
Sep 17 17:40:50.366414 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:50.366350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf" podStartSLOduration=8.478538825 podStartE2EDuration="26.366332396s" totalImagesPullingTime="17.887793571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:50.364779219 +0000 UTC m=+26.704172809" watchObservedRunningTime="2026-09-17 17:40:50.366332396 +0000 UTC m=+26.705725986"
Sep 17 17:40:51.040028 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.039991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:51.040855 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:51.040156    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:51.305454 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.305356    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a333d12-515a-4448-a372-27444bf5cb3f" containerID="04d6007dd977e8d1e8c7d350e219a4c7b859d81c6d2976865b7bf00dd52fd12a" exitCode=0
Sep 17 17:40:51.305659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.305460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerDied","Data":"04d6007dd977e8d1e8c7d350e219a4c7b859d81c6d2976865b7bf00dd52fd12a"}
Sep 17 17:40:51.306694 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.306442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:51.306694 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.306523    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:51.306694 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.306576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:51.322425 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.322396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:51.322745 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:51.322597    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:40:52.039642 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.039605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:52.039830 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.039605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:52.039830 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:52.039748    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:52.039830 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:52.039774    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:52.175818 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.175767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d67rh"]
Sep 17 17:40:52.176465 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.175904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:52.176465 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:52.176027    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:52.179239 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.179191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tnhn"]
Sep 17 17:40:52.181580 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.181534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z689h"]
Sep 17 17:40:52.313786 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.313756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:52.313929 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.313777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerStarted","Data":"daad26a76fa3a290dcad533bc0138c49758eb9c9e7a590b60b3b1e09cccafee0"}
Sep 17 17:40:52.313929 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:52.313866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:52.314041 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:52.314021    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:52.314387 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:52.314369    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:53.318055 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:53.318014    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a333d12-515a-4448-a372-27444bf5cb3f" containerID="daad26a76fa3a290dcad533bc0138c49758eb9c9e7a590b60b3b1e09cccafee0" exitCode=0
Sep 17 17:40:53.318535 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:53.318115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerDied","Data":"daad26a76fa3a290dcad533bc0138c49758eb9c9e7a590b60b3b1e09cccafee0"}
Sep 17 17:40:54.040424 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:54.040394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:54.040622 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:54.040443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:54.040622 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:54.040497    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:54.040622 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:54.040566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:54.040622 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:54.040570    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:54.040846 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:54.040670    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:54.323985 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:54.323951    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a333d12-515a-4448-a372-27444bf5cb3f" containerID="6ac21ab3d788f05fa24f2a66ca421ceff0991d8f2e4a562ed2a62bc9958352bf" exitCode=0
Sep 17 17:40:54.324422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:54.324036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerDied","Data":"6ac21ab3d788f05fa24f2a66ca421ceff0991d8f2e4a562ed2a62bc9958352bf"}
Sep 17 17:40:56.039944 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:56.039901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:56.040457 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:56.039911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:56.040457 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:56.040043    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z689h" podUID="df61c5cc-b897-402c-9cfd-1c7b02a8419f"
Sep 17 17:40:56.040457 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:56.040161    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d67rh" podUID="78c77eba-b3f9-4472-8dad-5b68b500e7dd"
Sep 17 17:40:56.040457 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:56.040207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:56.040457 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:56.040297    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tnhn" podUID="45cd1091-7a05-4050-9ea5-fc2c920202bf"
Sep 17 17:40:57.726490 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:57.726446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:57.726994 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.726636    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:57.726994 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.726715    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret podName:45cd1091-7a05-4050-9ea5-fc2c920202bf nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.726696285 +0000 UTC m=+66.066089874 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret") pod "global-pull-secret-syncer-2tnhn" (UID: "45cd1091-7a05-4050-9ea5-fc2c920202bf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:57.827469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:57.827359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:57.827657 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.827538    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:57.827657 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.827622    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.82760162 +0000 UTC m=+66.166995201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:57.927848 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:57.927811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:57.928047 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.928023    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:57.928120 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.928054    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:57.928120 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.928065    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dh9v6 for pod openshift-network-diagnostics/network-check-target-z689h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:57.928206 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:57.928132    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6 podName:df61c5cc-b897-402c-9cfd-1c7b02a8419f nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.928117705 +0000 UTC m=+66.267511273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh9v6" (UniqueName: "kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6") pod "network-check-target-z689h" (UID: "df61c5cc-b897-402c-9cfd-1c7b02a8419f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:57.984403 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:57.984372    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeReady"
Sep 17 17:40:57.984587 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:57.984551    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:40:58.026326 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.026269    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c4c4f9c74-szdr7"]
Sep 17 17:40:58.062764 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.062498    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-blzqz"]
Sep 17 17:40:58.062764 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.062740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.068349 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.068302    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:40:58.068552 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.068317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:40:58.069478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.068959    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mkt75\""
Sep 17 17:40:58.070219 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.070187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:40:58.075187 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.075150    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:40:58.084499 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.084446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.087526 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.087484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:40:58.087724 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.087487    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:40:58.087724 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.087499    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n5jl\""
Sep 17 17:40:58.090607 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.090571    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c4c4f9c74-szdr7"]
Sep 17 17:40:58.090861 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.090627    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j58hh"]
Sep 17 17:40:58.090861 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.090711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:40:58.090861 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.090737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:40:58.090861 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.090828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:40:58.093496 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.093465    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ccxk9\""
Sep 17 17:40:58.093738 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.093717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:40:58.093821 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.093739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7j64\""
Sep 17 17:40:58.093903 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.093882    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:40:58.093960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.093912    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:40:58.094211 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.094190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:40:58.109822 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.109789    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-blzqz"]
Sep 17 17:40:58.110000 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.109845    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j58hh"]
Sep 17 17:40:58.110000 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.109955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.113006 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.112983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:40:58.113161 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.112984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:40:58.113382 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.113362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xhmjd\""
Sep 17 17:40:58.113709 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.113678    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:40:58.230188 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-trusted-ca\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230188 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24a97cf2-225e-4096-9b32-05fe7c4f9e75-config-volume\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.230442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-image-registry-private-configuration\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-installation-pull-secrets\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-certificates\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25vxq\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-kube-api-access-25vxq\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-bound-sa-token\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230690 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1081e7e3-d38e-4fa8-827a-85928fbd4970-ca-trust-extracted\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.230690 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.230690 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqdc\" (UniqueName: \"kubernetes.io/projected/24a97cf2-225e-4096-9b32-05fe7c4f9e75-kube-api-access-qsqdc\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.230690 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd00e048-2992-4249-9ffd-d4d56bb6b446-cert\") pod \"ingress-canary-j58hh\" (UID: \"bd00e048-2992-4249-9ffd-d4d56bb6b446\") " pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.230690 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a97cf2-225e-4096-9b32-05fe7c4f9e75-tmp-dir\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.230690 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.230677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r828n\" (UniqueName: \"kubernetes.io/projected/bd00e048-2992-4249-9ffd-d4d56bb6b446-kube-api-access-r828n\") pod \"ingress-canary-j58hh\" (UID: \"bd00e048-2992-4249-9ffd-d4d56bb6b446\") " pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.331662 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1081e7e3-d38e-4fa8-827a-85928fbd4970-ca-trust-extracted\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.331662 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.331662 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqdc\" (UniqueName: \"kubernetes.io/projected/24a97cf2-225e-4096-9b32-05fe7c4f9e75-kube-api-access-qsqdc\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.331951 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd00e048-2992-4249-9ffd-d4d56bb6b446-cert\") pod \"ingress-canary-j58hh\" (UID: \"bd00e048-2992-4249-9ffd-d4d56bb6b446\") " pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.331951 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a97cf2-225e-4096-9b32-05fe7c4f9e75-tmp-dir\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.331951 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r828n\" (UniqueName: \"kubernetes.io/projected/bd00e048-2992-4249-9ffd-d4d56bb6b446-kube-api-access-r828n\") pod \"ingress-canary-j58hh\" (UID: \"bd00e048-2992-4249-9ffd-d4d56bb6b446\") " pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.331951 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-trusted-ca\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.331951 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.331817    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:40:58.331951 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24a97cf2-225e-4096-9b32-05fe7c4f9e75-config-volume\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.331963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-image-registry-private-configuration\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-installation-pull-secrets\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1081e7e3-d38e-4fa8-827a-85928fbd4970-ca-trust-extracted\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a97cf2-225e-4096-9b32-05fe7c4f9e75-tmp-dir\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-certificates\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.332144    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls podName:24a97cf2-225e-4096-9b32-05fe7c4f9e75 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:58.832121224 +0000 UTC m=+35.171514806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls") pod "dns-default-blzqz" (UID: "24a97cf2-225e-4096-9b32-05fe7c4f9e75") : secret "dns-default-metrics-tls" not found
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25vxq\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-kube-api-access-25vxq\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.332281 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-bound-sa-token\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.332727 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.332475    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:40:58.332727 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.332496    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4c4f9c74-szdr7: secret "image-registry-tls" not found
Sep 17 17:40:58.332727 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.332624    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls podName:1081e7e3-d38e-4fa8-827a-85928fbd4970 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:58.832592132 +0000 UTC m=+35.171985715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls") pod "image-registry-5c4c4f9c74-szdr7" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970") : secret "image-registry-tls" not found
Sep 17 17:40:58.332877 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.332732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-certificates\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.333285 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.333262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24a97cf2-225e-4096-9b32-05fe7c4f9e75-config-volume\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.333779 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.333756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-trusted-ca\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.337377 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.337320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-image-registry-private-configuration\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.337484 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.337393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-installation-pull-secrets\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.337484 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.337458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd00e048-2992-4249-9ffd-d4d56bb6b446-cert\") pod \"ingress-canary-j58hh\" (UID: \"bd00e048-2992-4249-9ffd-d4d56bb6b446\") " pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.346247 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.346217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqdc\" (UniqueName: \"kubernetes.io/projected/24a97cf2-225e-4096-9b32-05fe7c4f9e75-kube-api-access-qsqdc\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.346421 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.346394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r828n\" (UniqueName: \"kubernetes.io/projected/bd00e048-2992-4249-9ffd-d4d56bb6b446-kube-api-access-r828n\") pod \"ingress-canary-j58hh\" (UID: \"bd00e048-2992-4249-9ffd-d4d56bb6b446\") " pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.346750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.346725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25vxq\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-kube-api-access-25vxq\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.347327 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.347304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-bound-sa-token\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.427231 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.427192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j58hh"
Sep 17 17:40:58.838235 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.838015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:58.838635 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:58.838273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:58.838635 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.838186    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:40:58.838635 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.838427    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:40:58.838635 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.838438    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4c4f9c74-szdr7: secret "image-registry-tls" not found
Sep 17 17:40:58.838635 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.838438    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls podName:24a97cf2-225e-4096-9b32-05fe7c4f9e75 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.838419589 +0000 UTC m=+36.177813158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls") pod "dns-default-blzqz" (UID: "24a97cf2-225e-4096-9b32-05fe7c4f9e75") : secret "dns-default-metrics-tls" not found
Sep 17 17:40:58.838635 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:58.838488    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls podName:1081e7e3-d38e-4fa8-827a-85928fbd4970 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.838476371 +0000 UTC m=+36.177869939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls") pod "image-registry-5c4c4f9c74-szdr7" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970") : secret "image-registry-tls" not found
Sep 17 17:40:59.129320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.129240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j58hh"]
Sep 17 17:40:59.132994 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:40:59.132959    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd00e048_2992_4249_9ffd_d4d56bb6b446.slice/crio-b3a617f5744b78e4e244505d8801f13c062d1f584d702dbba0c20dcae0570470 WatchSource:0}: Error finding container b3a617f5744b78e4e244505d8801f13c062d1f584d702dbba0c20dcae0570470: Status 404 returned error can't find the container with id b3a617f5744b78e4e244505d8801f13c062d1f584d702dbba0c20dcae0570470
Sep 17 17:40:59.341530 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.341487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerStarted","Data":"45a8be331ea560920f9a4d1180894c51d869e4d382f6faea483285e96c516741"}
Sep 17 17:40:59.343364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.343309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j58hh" event={"ID":"bd00e048-2992-4249-9ffd-d4d56bb6b446","Type":"ContainerStarted","Data":"b3a617f5744b78e4e244505d8801f13c062d1f584d702dbba0c20dcae0570470"}
Sep 17 17:40:59.536528 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.536465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:59.661846 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.661808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:59.743151 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.743114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:59.845140 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.845103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:40:59.845793 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.845155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:40:59.845793 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:59.845254    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:40:59.845793 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:59.845256    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:40:59.845793 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:59.845335    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls podName:24a97cf2-225e-4096-9b32-05fe7c4f9e75 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:01.845315058 +0000 UTC m=+38.184708629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls") pod "dns-default-blzqz" (UID: "24a97cf2-225e-4096-9b32-05fe7c4f9e75") : secret "dns-default-metrics-tls" not found
Sep 17 17:40:59.845793 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:59.845267    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4c4f9c74-szdr7: secret "image-registry-tls" not found
Sep 17 17:40:59.845793 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:40:59.845425    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls podName:1081e7e3-d38e-4fa8-827a-85928fbd4970 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:01.845409755 +0000 UTC m=+38.184803347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls") pod "image-registry-5c4c4f9c74-szdr7" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970") : secret "image-registry-tls" not found
Sep 17 17:40:59.862439 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.862389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:59.971931 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:40:59.971837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:00.087474 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:00.087428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:00.349886 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:00.349793    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a333d12-515a-4448-a372-27444bf5cb3f" containerID="45a8be331ea560920f9a4d1180894c51d869e4d382f6faea483285e96c516741" exitCode=0
Sep 17 17:41:00.349886 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:00.349866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerDied","Data":"45a8be331ea560920f9a4d1180894c51d869e4d382f6faea483285e96c516741"}
Sep 17 17:41:00.510409 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:00.510371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:00.605811 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:00.605726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:00.703666 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:00.703603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:01.046756 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.046722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:01.119604 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.119568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:01.357107 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.356988    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a333d12-515a-4448-a372-27444bf5cb3f" containerID="0be3acbb92264b8005b0030c7c8f20c503207751506eb753a889ede865ddb03f" exitCode=0
Sep 17 17:41:01.357107 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.357079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerDied","Data":"0be3acbb92264b8005b0030c7c8f20c503207751506eb753a889ede865ddb03f"}
Sep 17 17:41:01.798152 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.798115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:01.865309 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.865267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:41:01.865513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.865337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:41:01.865513 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:01.865457    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:01.865513 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:01.865472    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:01.865513 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:01.865488    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4c4f9c74-szdr7: secret "image-registry-tls" not found
Sep 17 17:41:01.865726 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:01.865546    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls podName:24a97cf2-225e-4096-9b32-05fe7c4f9e75 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.865523045 +0000 UTC m=+42.204916614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls") pod "dns-default-blzqz" (UID: "24a97cf2-225e-4096-9b32-05fe7c4f9e75") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:01.865726 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:01.865567    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls podName:1081e7e3-d38e-4fa8-827a-85928fbd4970 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.865558389 +0000 UTC m=+42.204951958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls") pod "image-registry-5c4c4f9c74-szdr7" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970") : secret "image-registry-tls" not found
Sep 17 17:41:01.898275 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:01.898227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:02.362886 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:02.362841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz442" event={"ID":"4a333d12-515a-4448-a372-27444bf5cb3f","Type":"ContainerStarted","Data":"b3288003b95c7138cce1a21b8d94e774d75c3cfd753caeaca46abf617b0c6026"}
Sep 17 17:41:02.416752 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:02.416687    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tz442" podStartSLOduration=9.208119534 podStartE2EDuration="38.416661872s" totalImagesPullingTime="29.208542338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:02.416279306 +0000 UTC m=+38.755672897" watchObservedRunningTime="2026-09-17 17:41:02.416661872 +0000 UTC m=+38.756055464"
Sep 17 17:41:02.582617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:02.582575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:02.685500 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:02.685285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:03.110619 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:03.110578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:03.177431 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:03.177392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:03.367305 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:03.367212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j58hh" event={"ID":"bd00e048-2992-4249-9ffd-d4d56bb6b446","Type":"ContainerStarted","Data":"e3d5376cdbd098b58dfb7533f5d6e1767cc1f4107fe740bb89cd6ec466a4be15"}
Sep 17 17:41:03.386233 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:03.386182    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j58hh" podStartSLOduration=2.004448618 podStartE2EDuration="5.386164828s" totalImagesPullingTime="3.38171621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:03.385538518 +0000 UTC m=+39.724932112" watchObservedRunningTime="2026-09-17 17:41:03.386164828 +0000 UTC m=+39.725558419"
Sep 17 17:41:05.895868 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:05.895822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:41:05.896397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:05.895903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:41:05.896397 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:05.895998    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:05.896397 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:05.896073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls podName:24a97cf2-225e-4096-9b32-05fe7c4f9e75 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:13.896056475 +0000 UTC m=+50.235450074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls") pod "dns-default-blzqz" (UID: "24a97cf2-225e-4096-9b32-05fe7c4f9e75") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:05.896397 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:05.896009    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:05.896397 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:05.896127    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4c4f9c74-szdr7: secret "image-registry-tls" not found
Sep 17 17:41:05.896397 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:05.896206    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls podName:1081e7e3-d38e-4fa8-827a-85928fbd4970 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:13.896187437 +0000 UTC m=+50.235581028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls") pod "image-registry-5c4c4f9c74-szdr7" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970") : secret "image-registry-tls" not found
Sep 17 17:41:12.471305 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:12.471265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:12.625117 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:12.625056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:12.715382 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:12.715343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:12.816111 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:12.815982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:12.915511 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:12.915476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:13.961165 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:13.961119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:41:13.961632 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:13.961180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:41:13.961632 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:13.961291    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:13.961632 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:13.961293    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:13.961632 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:13.961304    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4c4f9c74-szdr7: secret "image-registry-tls" not found
Sep 17 17:41:13.961632 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:13.961383    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls podName:24a97cf2-225e-4096-9b32-05fe7c4f9e75 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.961359871 +0000 UTC m=+66.300753445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls") pod "dns-default-blzqz" (UID: "24a97cf2-225e-4096-9b32-05fe7c4f9e75") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:13.961632 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:13.961444    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls podName:1081e7e3-d38e-4fa8-827a-85928fbd4970 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.961430225 +0000 UTC m=+66.300823794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls") pod "image-registry-5c4c4f9c74-szdr7" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970") : secret "image-registry-tls" not found
Sep 17 17:41:20.611395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:20.611348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:20.673547 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:20.673504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:21.608291 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:21.608258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:21.727672 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:21.727638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.329284 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:23.329255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4mftf"
Sep 17 17:41:23.491560 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:23.491518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.593614 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:23.593511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.656412 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:23.656379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:25.627152 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.627117    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"]
Sep 17 17:41:25.630255 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.630231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.633150 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.633126    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:41:25.633283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.633211    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:41:25.633432 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.633418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.634233 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.634214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.634333 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.634264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nk9qs\""
Sep 17 17:41:25.641028 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.641005    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"]
Sep 17 17:41:25.649282 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.649257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87cf91a8-0c17-46f0-b87b-b8cce2936c96-serving-cert\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.649416 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.649328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87cf91a8-0c17-46f0-b87b-b8cce2936c96-config\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.649416 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.649389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcq9w\" (UniqueName: \"kubernetes.io/projected/87cf91a8-0c17-46f0-b87b-b8cce2936c96-kube-api-access-mcq9w\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.750829 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.750788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87cf91a8-0c17-46f0-b87b-b8cce2936c96-serving-cert\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.751057 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.750857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87cf91a8-0c17-46f0-b87b-b8cce2936c96-config\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.751057 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.750884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcq9w\" (UniqueName: \"kubernetes.io/projected/87cf91a8-0c17-46f0-b87b-b8cce2936c96-kube-api-access-mcq9w\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.751473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.751447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87cf91a8-0c17-46f0-b87b-b8cce2936c96-config\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.754160 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.754138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87cf91a8-0c17-46f0-b87b-b8cce2936c96-serving-cert\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.769864 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.769834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcq9w\" (UniqueName: \"kubernetes.io/projected/87cf91a8-0c17-46f0-b87b-b8cce2936c96-kube-api-access-mcq9w\") pod \"service-ca-operator-57cdc4b768-gs2qg\" (UID: \"87cf91a8-0c17-46f0-b87b-b8cce2936c96\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.796398 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.796288    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"]
Sep 17 17:41:25.799226 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.799203    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"]
Sep 17 17:41:25.799354 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.799321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:25.801260 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.801239    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"]
Sep 17 17:41:25.801378 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.801361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.802147 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.802128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.802495 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.802478    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:41:25.802708 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.802691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.803556 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.803536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"
Sep 17 17:41:25.804203 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.804187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-msc7l\""
Sep 17 17:41:25.804974 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.804951    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:41:25.805076 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.804984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.806453 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.806432    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.806607 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.806478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:41:25.806945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.806927    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.807746 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.807731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fgpld\""
Sep 17 17:41:25.808357 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.808344    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.809140 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.809124    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-945m6\""
Sep 17 17:41:25.815496 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.815447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"]
Sep 17 17:41:25.816577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.816552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"]
Sep 17 17:41:25.820336 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.820170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"]
Sep 17 17:41:25.851340 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.851296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:25.851340 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.851336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9kl\" (UniqueName: \"kubernetes.io/projected/19dd41c1-e0dc-4dee-962f-ee14a11d0438-kube-api-access-7g9kl\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:25.851643 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.851471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4tl\" (UniqueName: \"kubernetes.io/projected/a0c85c0e-7126-471d-bd1f-292693933089-kube-api-access-sd4tl\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.851643 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.851518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.851643 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.851580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0c85c0e-7126-471d-bd1f-292693933089-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.851643 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.851633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7272\" (UniqueName: \"kubernetes.io/projected/36932b3a-b41e-4d15-b034-170f2ea4c6a8-kube-api-access-s7272\") pod \"volume-data-source-validator-5c4db8bdf9-mfj5q\" (UID: \"36932b3a-b41e-4d15-b034-170f2ea4c6a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"
Sep 17 17:41:25.852180 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.852145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jnfbk"]
Sep 17 17:41:25.854731 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.854715    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"]
Sep 17 17:41:25.854846 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.854829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:25.857134 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.857109    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"]
Sep 17 17:41:25.857238 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.857201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"
Sep 17 17:41:25.858045 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.858025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.858150 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.858049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.858220 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.858186    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:41:25.858365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.858346    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:41:25.858550 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.858535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-75f24\""
Sep 17 17:41:25.859386 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.859367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:25.864038 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.864015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:41:25.864188 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.864111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rcnkv\""
Sep 17 17:41:25.864188 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.864157    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:41:25.864188 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.864157    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.864614 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.864593    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8x5jd\""
Sep 17 17:41:25.865541 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.865523    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.866264 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.866246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:41:25.869507 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.869486    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79d7b547b8-gfldz"]
Sep 17 17:41:25.872300 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.872282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:25.876198 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.876173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"]
Sep 17 17:41:25.877105 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.877063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"]
Sep 17 17:41:25.879127 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.879043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:41:25.879239 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.879154    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.879239 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.879048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:41:25.879373 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.879297    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:41:25.879373 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.879333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:41:25.879488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.879426    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tccz6\""
Sep 17 17:41:25.879488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.879466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.892014 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.891987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jnfbk"]
Sep 17 17:41:25.892995 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.892968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79d7b547b8-gfldz"]
Sep 17 17:41:25.939106 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.939052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"
Sep 17 17:41:25.952746 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.952717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qh4w\" (UniqueName: \"kubernetes.io/projected/f6481e0f-0af5-42c7-86ac-0c268a08bb19-kube-api-access-6qh4w\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:25.952876 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.952759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7272\" (UniqueName: \"kubernetes.io/projected/36932b3a-b41e-4d15-b034-170f2ea4c6a8-kube-api-access-s7272\") pod \"volume-data-source-validator-5c4db8bdf9-mfj5q\" (UID: \"36932b3a-b41e-4d15-b034-170f2ea4c6a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"
Sep 17 17:41:25.952876 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.952780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:25.952876 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.952809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-default-certificate\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:25.952983 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.952936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9kl\" (UniqueName: \"kubernetes.io/projected/19dd41c1-e0dc-4dee-962f-ee14a11d0438-kube-api-access-7g9kl\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:25.953027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.952979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/feae8d3b-b661-4b65-8713-fbf8404a86ed-serving-cert\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:25.953027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/feae8d3b-b661-4b65-8713-fbf8404a86ed-config\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:25.953139 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmd79\" (UniqueName: \"kubernetes.io/projected/feae8d3b-b661-4b65-8713-fbf8404a86ed-kube-api-access-fmd79\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:25.953202 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feae8d3b-b661-4b65-8713-fbf8404a86ed-trusted-ca\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:25.953247 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6s9\" (UniqueName: \"kubernetes.io/projected/1f9ff7dd-7548-4361-bf51-dc1b158e530c-kube-api-access-ll6s9\") pod \"network-check-source-7cbc45fddc-g6wht\" (UID: \"1f9ff7dd-7548-4361-bf51-dc1b158e530c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"
Sep 17 17:41:25.953294 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcgw9\" (UniqueName: \"kubernetes.io/projected/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-kube-api-access-lcgw9\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:25.953333 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6481e0f-0af5-42c7-86ac-0c268a08bb19-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:25.953372 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6481e0f-0af5-42c7-86ac-0c268a08bb19-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:25.953414 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:25.953456 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:25.953493 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0c85c0e-7126-471d-bd1f-292693933089-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.953536 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4tl\" (UniqueName: \"kubernetes.io/projected/a0c85c0e-7126-471d-bd1f-292693933089-kube-api-access-sd4tl\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.953585 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.953635 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:25.953571    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:41:25.953743 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:25.953715    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:25.953839 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.953578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-stats-auth\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:25.953839 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:25.953759    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls podName:19dd41c1-e0dc-4dee-962f-ee14a11d0438 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.453691403 +0000 UTC m=+62.793085005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8n6t" (UID: "19dd41c1-e0dc-4dee-962f-ee14a11d0438") : secret "samples-operator-tls" not found
Sep 17 17:41:25.953839 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:25.953793    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls podName:a0c85c0e-7126-471d-bd1f-292693933089 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.45377346 +0000 UTC m=+62.793167044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xqcxc" (UID: "a0c85c0e-7126-471d-bd1f-292693933089") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:25.954571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.954552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a0c85c0e-7126-471d-bd1f-292693933089-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.976075 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.976038    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gczxt"]
Sep 17 17:41:25.976283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.976186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9kl\" (UniqueName: \"kubernetes.io/projected/19dd41c1-e0dc-4dee-962f-ee14a11d0438-kube-api-access-7g9kl\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:25.976283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.976204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4tl\" (UniqueName: \"kubernetes.io/projected/a0c85c0e-7126-471d-bd1f-292693933089-kube-api-access-sd4tl\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:25.976870 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.976837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7272\" (UniqueName: \"kubernetes.io/projected/36932b3a-b41e-4d15-b034-170f2ea4c6a8-kube-api-access-s7272\") pod \"volume-data-source-validator-5c4db8bdf9-mfj5q\" (UID: \"36932b3a-b41e-4d15-b034-170f2ea4c6a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"
Sep 17 17:41:25.981881 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.981856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:25.995409 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.995379    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.995545 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.995379    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:41:25.995545 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.995484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rlvjv\""
Sep 17 17:41:25.995545 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:25.995525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:41:26.013909 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.013881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:41:26.020523 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.020497    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:41:26.043671 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.043645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gczxt"]
Sep 17 17:41:26.052379 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.052347    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"]
Sep 17 17:41:26.054636 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6s9\" (UniqueName: \"kubernetes.io/projected/1f9ff7dd-7548-4361-bf51-dc1b158e530c-kube-api-access-ll6s9\") pod \"network-check-source-7cbc45fddc-g6wht\" (UID: \"1f9ff7dd-7548-4361-bf51-dc1b158e530c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"
Sep 17 17:41:26.054731 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcgw9\" (UniqueName: \"kubernetes.io/projected/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-kube-api-access-lcgw9\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.054731 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6481e0f-0af5-42c7-86ac-0c268a08bb19-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:26.054808 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6481e0f-0af5-42c7-86ac-0c268a08bb19-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:26.054808 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.054886 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-serving-cert\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.054886 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-snapshots\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.054972 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.054926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-stats-auth\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.054972 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.054960    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.554937325 +0000 UTC m=+62.894330914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:41:26.055076 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.055133 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.055179 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z42j\" (UniqueName: \"kubernetes.io/projected/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-kube-api-access-8z42j\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.055220 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qh4w\" (UniqueName: \"kubernetes.io/projected/f6481e0f-0af5-42c7-86ac-0c268a08bb19-kube-api-access-6qh4w\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:26.055284 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.055327 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:26.055327 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-default-certificate\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.055432 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/feae8d3b-b661-4b65-8713-fbf8404a86ed-serving-cert\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.055432 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6481e0f-0af5-42c7-86ac-0c268a08bb19-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:26.055432 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.055429    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:41:26.055578 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.055474    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.555460914 +0000 UTC m=+62.894854486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : secret "router-metrics-certs-default" not found
Sep 17 17:41:26.055578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-tmp\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.055578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/feae8d3b-b661-4b65-8713-fbf8404a86ed-config\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.055749 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmd79\" (UniqueName: \"kubernetes.io/projected/feae8d3b-b661-4b65-8713-fbf8404a86ed-kube-api-access-fmd79\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.055749 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.055648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feae8d3b-b661-4b65-8713-fbf8404a86ed-trusted-ca\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.056421 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.056394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/feae8d3b-b661-4b65-8713-fbf8404a86ed-config\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.056989 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.056965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feae8d3b-b661-4b65-8713-fbf8404a86ed-trusted-ca\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.058172 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.058144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-default-certificate\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.058172 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.058166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-stats-auth\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.058340 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.058323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6481e0f-0af5-42c7-86ac-0c268a08bb19-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:26.058448 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.058432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/feae8d3b-b661-4b65-8713-fbf8404a86ed-serving-cert\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.061442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.061419    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:41:26.061442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.061435    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:41:26.061587 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.061565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9gvkz\""
Sep 17 17:41:26.073173 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.073142    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"]
Sep 17 17:41:26.077394 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.077372    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:41:26.078121 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.078097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qh4w\" (UniqueName: \"kubernetes.io/projected/f6481e0f-0af5-42c7-86ac-0c268a08bb19-kube-api-access-6qh4w\") pod \"kube-storage-version-migrator-operator-55889f7b47-dmz2q\" (UID: \"f6481e0f-0af5-42c7-86ac-0c268a08bb19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:26.080294 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.080275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmd79\" (UniqueName: \"kubernetes.io/projected/feae8d3b-b661-4b65-8713-fbf8404a86ed-kube-api-access-fmd79\") pod \"console-operator-fd4bb46c-jnfbk\" (UID: \"feae8d3b-b661-4b65-8713-fbf8404a86ed\") " pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.087758 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.087739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcgw9\" (UniqueName: \"kubernetes.io/projected/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-kube-api-access-lcgw9\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.089346 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.089325    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:41:26.098905 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.098883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6s9\" (UniqueName: \"kubernetes.io/projected/1f9ff7dd-7548-4361-bf51-dc1b158e530c-kube-api-access-ll6s9\") pod \"network-check-source-7cbc45fddc-g6wht\" (UID: \"1f9ff7dd-7548-4361-bf51-dc1b158e530c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"
Sep 17 17:41:26.122938 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.122902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"
Sep 17 17:41:26.156491 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-tmp\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.156491 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/44df5c8b-3917-43fa-b640-104adae7fec0-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:26.156701 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-serving-cert\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.156701 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-snapshots\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.156701 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:26.156794 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.156794 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.156945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z42j\" (UniqueName: \"kubernetes.io/projected/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-kube-api-access-8z42j\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.156945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.156937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-tmp\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.157397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.157372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.157397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.157389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-snapshots\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.157752 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.157731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.159206 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.159185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-serving-cert\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.165338 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.165307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:26.166460 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.166437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z42j\" (UniqueName: \"kubernetes.io/projected/f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6-kube-api-access-8z42j\") pod \"insights-operator-6f9c644dd9-gczxt\" (UID: \"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6\") " pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.172329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.172298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"
Sep 17 17:41:26.179217 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.179190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"
Sep 17 17:41:26.199740 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.199644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:26.222478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.222359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg"]
Sep 17 17:41:26.226832 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:26.226772    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cf91a8_0c17_46f0_b87b_b8cce2936c96.slice/crio-eb4968d908adc53b41380852d1c09c3f33c0270dd8ea8a9d2f22cebbb9575027 WatchSource:0}: Error finding container eb4968d908adc53b41380852d1c09c3f33c0270dd8ea8a9d2f22cebbb9575027: Status 404 returned error can't find the container with id eb4968d908adc53b41380852d1c09c3f33c0270dd8ea8a9d2f22cebbb9575027
Sep 17 17:41:26.258213 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.258182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/44df5c8b-3917-43fa-b640-104adae7fec0-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:26.258373 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.258287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:26.258480 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.258456    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:41:26.258655 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.258631    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert podName:44df5c8b-3917-43fa-b640-104adae7fec0 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.758545131 +0000 UTC m=+63.097938717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fb6ww" (UID: "44df5c8b-3917-43fa-b640-104adae7fec0") : secret "networking-console-plugin-cert" not found
Sep 17 17:41:26.258941 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.258925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/44df5c8b-3917-43fa-b640-104adae7fec0-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:26.294472 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.294432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-gczxt"
Sep 17 17:41:26.301715 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.301688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:26.413744 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.413709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q"]
Sep 17 17:41:26.416789 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:26.416760    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36932b3a_b41e_4d15_b034_170f2ea4c6a8.slice/crio-80684bb38b548b52c810aae2116f66c28ed97757a8e96e451f7c5745ff6ca6ba WatchSource:0}: Error finding container 80684bb38b548b52c810aae2116f66c28ed97757a8e96e451f7c5745ff6ca6ba: Status 404 returned error can't find the container with id 80684bb38b548b52c810aae2116f66c28ed97757a8e96e451f7c5745ff6ca6ba
Sep 17 17:41:26.427155 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.427121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q" event={"ID":"36932b3a-b41e-4d15-b034-170f2ea4c6a8","Type":"ContainerStarted","Data":"80684bb38b548b52c810aae2116f66c28ed97757a8e96e451f7c5745ff6ca6ba"}
Sep 17 17:41:26.428166 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.428143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg" event={"ID":"87cf91a8-0c17-46f0-b87b-b8cce2936c96","Type":"ContainerStarted","Data":"eb4968d908adc53b41380852d1c09c3f33c0270dd8ea8a9d2f22cebbb9575027"}
Sep 17 17:41:26.442535 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.442502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:26.460376 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.460338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:26.460513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.460424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:26.460513 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.460498    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:41:26.460602 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.460569    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls podName:19dd41c1-e0dc-4dee-962f-ee14a11d0438 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.460543389 +0000 UTC m=+63.799936970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8n6t" (UID: "19dd41c1-e0dc-4dee-962f-ee14a11d0438") : secret "samples-operator-tls" not found
Sep 17 17:41:26.460602 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.460567    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:26.460602 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.460597    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls podName:a0c85c0e-7126-471d-bd1f-292693933089 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.46059085 +0000 UTC m=+63.799984431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xqcxc" (UID: "a0c85c0e-7126-471d-bd1f-292693933089") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:26.470751 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.470720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jnfbk"]
Sep 17 17:41:26.474137 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:26.474109    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeae8d3b_b661_4b65_8713_fbf8404a86ed.slice/crio-7dfd2ec15e1b3beb0139eab8313826a13d2138426caa9190083562299e998041 WatchSource:0}: Error finding container 7dfd2ec15e1b3beb0139eab8313826a13d2138426caa9190083562299e998041: Status 404 returned error can't find the container with id 7dfd2ec15e1b3beb0139eab8313826a13d2138426caa9190083562299e998041
Sep 17 17:41:26.492957 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.492916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q"]
Sep 17 17:41:26.494437 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:26.494221    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6481e0f_0af5_42c7_86ac_0c268a08bb19.slice/crio-b9d222ee5cf6346b12a1e63c02fb977e9cf459bd55f7fb57f0e872a754ded3b1 WatchSource:0}: Error finding container b9d222ee5cf6346b12a1e63c02fb977e9cf459bd55f7fb57f0e872a754ded3b1: Status 404 returned error can't find the container with id b9d222ee5cf6346b12a1e63c02fb977e9cf459bd55f7fb57f0e872a754ded3b1
Sep 17 17:41:26.495189 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.495141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht"]
Sep 17 17:41:26.561682 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.561641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.561887 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.561727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:26.561887 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.561818    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:41:26.561887 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.561861    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.561846895 +0000 UTC m=+63.901240464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:41:26.561887 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.561889    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.561871449 +0000 UTC m=+63.901265031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : secret "router-metrics-certs-default" not found
Sep 17 17:41:26.585854 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.585819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gczxt"]
Sep 17 17:41:26.589342 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:26.589317    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1331c2d_f21b_4bfa_a2a2_af5217bb1ef6.slice/crio-2aef73e452d5dc5806b3f4bc78329e378331c70c4793230b5e1690c153311312 WatchSource:0}: Error finding container 2aef73e452d5dc5806b3f4bc78329e378331c70c4793230b5e1690c153311312: Status 404 returned error can't find the container with id 2aef73e452d5dc5806b3f4bc78329e378331c70c4793230b5e1690c153311312
Sep 17 17:41:26.763629 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:26.763572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:26.764116 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.763751    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:41:26.764116 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:26.763832    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert podName:44df5c8b-3917-43fa-b640-104adae7fec0 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.763811663 +0000 UTC m=+64.103205253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fb6ww" (UID: "44df5c8b-3917-43fa-b640-104adae7fec0") : secret "networking-console-plugin-cert" not found
Sep 17 17:41:27.435703 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.435617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q" event={"ID":"f6481e0f-0af5-42c7-86ac-0c268a08bb19","Type":"ContainerStarted","Data":"b9d222ee5cf6346b12a1e63c02fb977e9cf459bd55f7fb57f0e872a754ded3b1"}
Sep 17 17:41:27.439509 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.439407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" event={"ID":"feae8d3b-b661-4b65-8713-fbf8404a86ed","Type":"ContainerStarted","Data":"7dfd2ec15e1b3beb0139eab8313826a13d2138426caa9190083562299e998041"}
Sep 17 17:41:27.443438 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.443202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gczxt" event={"ID":"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6","Type":"ContainerStarted","Data":"2aef73e452d5dc5806b3f4bc78329e378331c70c4793230b5e1690c153311312"}
Sep 17 17:41:27.447250 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.447150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht" event={"ID":"1f9ff7dd-7548-4361-bf51-dc1b158e530c","Type":"ContainerStarted","Data":"1a530928f02b14378c1bfc39262f5bc10c1d794c9e604c8dec31bec533b63930"}
Sep 17 17:41:27.472589 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.471024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:27.472589 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.471240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:27.472589 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.471580    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:27.472589 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.471807    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:41:27.472589 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.471896    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls podName:19dd41c1-e0dc-4dee-962f-ee14a11d0438 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.471873977 +0000 UTC m=+65.811267560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8n6t" (UID: "19dd41c1-e0dc-4dee-962f-ee14a11d0438") : secret "samples-operator-tls" not found
Sep 17 17:41:27.473483 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.473439    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls podName:a0c85c0e-7126-471d-bd1f-292693933089 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.473397047 +0000 UTC m=+65.812790629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xqcxc" (UID: "a0c85c0e-7126-471d-bd1f-292693933089") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:27.572645 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.572593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:27.572806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.572716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:27.572874 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.572827    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:41:27.572914 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.572869    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.572851221 +0000 UTC m=+65.912244808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:41:27.572914 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.572892    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.572882344 +0000 UTC m=+65.912275913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : secret "router-metrics-certs-default" not found
Sep 17 17:41:27.775672 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:27.775566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:27.776200 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.775849    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:41:27.776200 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:27.775918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert podName:44df5c8b-3917-43fa-b640-104adae7fec0 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:29.77589819 +0000 UTC m=+66.115291763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fb6ww" (UID: "44df5c8b-3917-43fa-b640-104adae7fec0") : secret "networking-console-plugin-cert" not found
Sep 17 17:41:29.496655 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.496567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:29.497297 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.496743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:29.497297 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.496829    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:41:29.497297 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.496958    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls podName:19dd41c1-e0dc-4dee-962f-ee14a11d0438 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.496927239 +0000 UTC m=+69.836320810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8n6t" (UID: "19dd41c1-e0dc-4dee-962f-ee14a11d0438") : secret "samples-operator-tls" not found
Sep 17 17:41:29.497297 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.497019    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:29.497297 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.497076    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls podName:a0c85c0e-7126-471d-bd1f-292693933089 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.497058251 +0000 UTC m=+69.836451823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xqcxc" (UID: "a0c85c0e-7126-471d-bd1f-292693933089") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:29.598034 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.597970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:29.598237 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.598150    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.598122276 +0000 UTC m=+69.937515862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:41:29.598237 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.598207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:29.598403 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.598378    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:41:29.598525 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.598440    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.598421904 +0000 UTC m=+69.937815478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : secret "router-metrics-certs-default" not found
Sep 17 17:41:29.800975 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.800873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:41:29.800975 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.800939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:29.801182 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.801062    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:41:29.801182 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.801143    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert podName:44df5c8b-3917-43fa-b640-104adae7fec0 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.801126079 +0000 UTC m=+70.140519667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fb6ww" (UID: "44df5c8b-3917-43fa-b640-104adae7fec0") : secret "networking-console-plugin-cert" not found
Sep 17 17:41:29.804142 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.804115    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:41:29.814225 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.814190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45cd1091-7a05-4050-9ea5-fc2c920202bf-original-pull-secret\") pod \"global-pull-secret-syncer-2tnhn\" (UID: \"45cd1091-7a05-4050-9ea5-fc2c920202bf\") " pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:41:29.902272 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.902229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:41:29.905551 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.905519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:41:29.912992 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.912955    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:41:29.913190 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:29.913049    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs podName:78c77eba-b3f9-4472-8dad-5b68b500e7dd nodeName:}" failed. No retries permitted until 2026-09-17 17:42:33.913025505 +0000 UTC m=+130.252419087 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs") pod "network-metrics-daemon-d67rh" (UID: "78c77eba-b3f9-4472-8dad-5b68b500e7dd") : secret "metrics-daemon-secret" not found
Sep 17 17:41:29.921521 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:29.921483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tnhn"
Sep 17 17:41:30.003635 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:30.003598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:41:30.003857 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:30.003674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:41:30.003857 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:30.003704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:41:30.003857 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:30.003783    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:30.003857 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:30.003807    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4c4f9c74-szdr7: secret "image-registry-tls" not found
Sep 17 17:41:30.003857 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:30.003823    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:30.004078 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:30.003886    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls podName:1081e7e3-d38e-4fa8-827a-85928fbd4970 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:02.003862034 +0000 UTC m=+98.343255620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls") pod "image-registry-5c4c4f9c74-szdr7" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970") : secret "image-registry-tls" not found
Sep 17 17:41:30.004078 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:30.003906    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls podName:24a97cf2-225e-4096-9b32-05fe7c4f9e75 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:02.003896138 +0000 UTC m=+98.343289709 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls") pod "dns-default-blzqz" (UID: "24a97cf2-225e-4096-9b32-05fe7c4f9e75") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:30.006917 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:30.006886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9v6\" (UniqueName: \"kubernetes.io/projected/df61c5cc-b897-402c-9cfd-1c7b02a8419f-kube-api-access-dh9v6\") pod \"network-check-target-z689h\" (UID: \"df61c5cc-b897-402c-9cfd-1c7b02a8419f\") " pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:41:30.211125 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:30.211069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ccxk9\""
Sep 17 17:41:30.217694 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:30.217666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:41:31.471915 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.471872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q" event={"ID":"36932b3a-b41e-4d15-b034-170f2ea4c6a8","Type":"ContainerStarted","Data":"cce0164c8a2f6b52ac90429c8dd121e4315ae9aa831bcebdce531804c16a657a"}
Sep 17 17:41:31.475905 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.475126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gczxt" event={"ID":"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6","Type":"ContainerStarted","Data":"5f4a23fea8093f802d576d799aeced2596062f809f40f298a8bb414bf17fb8f5"}
Sep 17 17:41:31.478808 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.478774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht" event={"ID":"1f9ff7dd-7548-4361-bf51-dc1b158e530c","Type":"ContainerStarted","Data":"1ee1dd9e35f47cabaec3e1f0b5dcfc96005b1ab84011389dab4993cd4a91894f"}
Sep 17 17:41:31.481337 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.480590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg" event={"ID":"87cf91a8-0c17-46f0-b87b-b8cce2936c96","Type":"ContainerStarted","Data":"de77ddd565a063c219be116bd44fa87094c1562bd943570a7011fa3eb53bfd7d"}
Sep 17 17:41:31.482721 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.482691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q" event={"ID":"f6481e0f-0af5-42c7-86ac-0c268a08bb19","Type":"ContainerStarted","Data":"864f7effab4e9d9d9b893a07eff2c601a2b3fcd8663e38a1d03ae05a0304861e"}
Sep 17 17:41:31.484360 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.484336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" event={"ID":"feae8d3b-b661-4b65-8713-fbf8404a86ed","Type":"ContainerStarted","Data":"3e983ec51eb8032eeddef9d808c0a98b6a5bb377081855946a9ca7eb39c6d227"}
Sep 17 17:41:31.484743 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.484726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:31.486034 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.486008    2580 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jnfbk container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Sep 17 17:41:31.486137 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.486060    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" podUID="feae8d3b-b661-4b65-8713-fbf8404a86ed" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Sep 17 17:41:31.511374 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.511250    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mfj5q" podStartSLOduration=1.687701923 podStartE2EDuration="6.511227927s" totalImagesPullingTime="4.823526004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.492296236 +0000 UTC m=+67.831689829" watchObservedRunningTime="2026-09-17 17:41:31.511227927 +0000 UTC m=+67.850621512"
Sep 17 17:41:31.536290 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.536208    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg" podStartSLOduration=1.492245184 podStartE2EDuration="6.536183356s" totalImagesPullingTime="5.043938172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.510969057 +0000 UTC m=+67.850362649" watchObservedRunningTime="2026-09-17 17:41:31.536183356 +0000 UTC m=+67.875576944"
Sep 17 17:41:31.539007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.537699    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g6wht" podStartSLOduration=1.7499636600000001 podStartE2EDuration="6.537670621s" totalImagesPullingTime="4.787706961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.532272442 +0000 UTC m=+67.871666033" watchObservedRunningTime="2026-09-17 17:41:31.537670621 +0000 UTC m=+67.877064213"
Sep 17 17:41:31.539007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.538122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z689h"]
Sep 17 17:41:31.558755 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.558695    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-gczxt" podStartSLOduration=1.875886186 podStartE2EDuration="6.558671573s" totalImagesPullingTime="4.682785387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.557278465 +0000 UTC m=+67.896672058" watchObservedRunningTime="2026-09-17 17:41:31.558671573 +0000 UTC m=+67.898065167"
Sep 17 17:41:31.577803 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.577522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tnhn"]
Sep 17 17:41:31.586401 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.586304    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" podStartSLOduration=1.788420592 podStartE2EDuration="6.586262524s" totalImagesPullingTime="4.797841932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.584920545 +0000 UTC m=+67.924314137" watchObservedRunningTime="2026-09-17 17:41:31.586262524 +0000 UTC m=+67.925656115"
Sep 17 17:41:31.610742 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.610686    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q" podStartSLOduration=1.834659463 podStartE2EDuration="6.610660926s" totalImagesPullingTime="4.776001463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.609377477 +0000 UTC m=+67.948771068" watchObservedRunningTime="2026-09-17 17:41:31.610660926 +0000 UTC m=+67.950054519"
Sep 17 17:41:31.666382 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.666348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:31.769709 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:31.769611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.490884 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.490779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z689h" event={"ID":"df61c5cc-b897-402c-9cfd-1c7b02a8419f","Type":"ContainerStarted","Data":"2bc832ae3385c4665ba5e4b6e41852ba039cc2aa9573d68da2d7f18c8d1709f9"}
Sep 17 17:41:32.490884 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.490843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z689h" event={"ID":"df61c5cc-b897-402c-9cfd-1c7b02a8419f","Type":"ContainerStarted","Data":"9bb0c1a5f5e8a2506d4d278656b6d065be9615ff2f007f997df4e9956b8b418b"}
Sep 17 17:41:32.491577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.491515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:41:32.493497 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.493416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tnhn" event={"ID":"45cd1091-7a05-4050-9ea5-fc2c920202bf","Type":"ContainerStarted","Data":"8c4a58d4e1e437c3fab5a5de86ea7dc052c023b4794bbd3f88f02f66220b069c"}
Sep 17 17:41:32.496710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.495495    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/0.log"
Sep 17 17:41:32.496710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.495535    2580 generic.go:356] "Generic (PLEG): container finished" podID="feae8d3b-b661-4b65-8713-fbf8404a86ed" containerID="3e983ec51eb8032eeddef9d808c0a98b6a5bb377081855946a9ca7eb39c6d227" exitCode=255
Sep 17 17:41:32.496710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.496366    2580 scope.go:117] "RemoveContainer" containerID="3e983ec51eb8032eeddef9d808c0a98b6a5bb377081855946a9ca7eb39c6d227"
Sep 17 17:41:32.496710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.496453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" event={"ID":"feae8d3b-b661-4b65-8713-fbf8404a86ed","Type":"ContainerDied","Data":"3e983ec51eb8032eeddef9d808c0a98b6a5bb377081855946a9ca7eb39c6d227"}
Sep 17 17:41:32.515739 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.515685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z689h" podStartSLOduration=68.515658174 podStartE2EDuration="1m8.515658174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:32.515335956 +0000 UTC m=+68.854729548" watchObservedRunningTime="2026-09-17 17:41:32.515658174 +0000 UTC m=+68.855051768"
Sep 17 17:41:32.719271 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.719214    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.850522 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.850369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.979575 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.979534    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7dtsh"]
Sep 17 17:41:32.994779 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:32.994741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.003685 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.003649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.015553 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.015519    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:41:33.015773 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.015750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zrqgb\""
Sep 17 17:41:33.026444 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.026403    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:41:33.031105 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.031042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7dtsh"]
Sep 17 17:41:33.140293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.140160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2f72097-38a0-4d9c-bace-cffca394ce07-crio-socket\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.140293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.140256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wvf\" (UniqueName: \"kubernetes.io/projected/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-api-access-24wvf\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.140293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.140288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.140582 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.140323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.140582 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.140384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f72097-38a0-4d9c-bace-cffca394ce07-data-volume\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.199061 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.199019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.241500 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.241450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wvf\" (UniqueName: \"kubernetes.io/projected/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-api-access-24wvf\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.241764 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.241509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.241764 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.241540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.241764 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.241576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f72097-38a0-4d9c-bace-cffca394ce07-data-volume\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.241928 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.241775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2f72097-38a0-4d9c-bace-cffca394ce07-crio-socket\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.241928 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.241892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2f72097-38a0-4d9c-bace-cffca394ce07-crio-socket\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.242037 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.242011    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.242142 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.242080    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:33.242142 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.242149    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.742115683 +0000 UTC m=+70.081509265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.242366 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.242194    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:33.742169212 +0000 UTC m=+70.081562795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:33.242640 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.242595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f72097-38a0-4d9c-bace-cffca394ce07-data-volume\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.252356 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.252315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wvf\" (UniqueName: \"kubernetes.io/projected/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-api-access-24wvf\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.299218 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.299158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.473591 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.473546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.502956 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.502920    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:41:33.503593 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.503571    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/0.log"
Sep 17 17:41:33.503725 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.503615    2580 generic.go:356] "Generic (PLEG): container finished" podID="feae8d3b-b661-4b65-8713-fbf8404a86ed" containerID="6fc2f027ad2e6d4fe638bbddec425d77dcb8e2fa2f8cdafa087c440a2c794de3" exitCode=255
Sep 17 17:41:33.504211 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.503869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" event={"ID":"feae8d3b-b661-4b65-8713-fbf8404a86ed","Type":"ContainerDied","Data":"6fc2f027ad2e6d4fe638bbddec425d77dcb8e2fa2f8cdafa087c440a2c794de3"}
Sep 17 17:41:33.504211 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.504006    2580 scope.go:117] "RemoveContainer" containerID="3e983ec51eb8032eeddef9d808c0a98b6a5bb377081855946a9ca7eb39c6d227"
Sep 17 17:41:33.504621 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.504469    2580 scope.go:117] "RemoveContainer" containerID="6fc2f027ad2e6d4fe638bbddec425d77dcb8e2fa2f8cdafa087c440a2c794de3"
Sep 17 17:41:33.504730 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.504664    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jnfbk_openshift-console-operator(feae8d3b-b661-4b65-8713-fbf8404a86ed)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" podUID="feae8d3b-b661-4b65-8713-fbf8404a86ed"
Sep 17 17:41:33.544782 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.544745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:33.544937 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.544912    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:33.545010 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.544911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:33.545010 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.544990    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:41:33.545114 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.544995    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls podName:a0c85c0e-7126-471d-bd1f-292693933089 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:41.544962445 +0000 UTC m=+77.884356033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xqcxc" (UID: "a0c85c0e-7126-471d-bd1f-292693933089") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:33.545114 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.545040    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls podName:19dd41c1-e0dc-4dee-962f-ee14a11d0438 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:41.545029052 +0000 UTC m=+77.884422623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8n6t" (UID: "19dd41c1-e0dc-4dee-962f-ee14a11d0438") : secret "samples-operator-tls" not found
Sep 17 17:41:33.568293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.568240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.645750 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.645703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:33.645980 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.645946    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:41.645919119 +0000 UTC m=+77.985312711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:41:33.646136 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.646055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:33.646545 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.646255    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:41:33.646545 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.646338    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:41.64632193 +0000 UTC m=+77.985715524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : secret "router-metrics-certs-default" not found
Sep 17 17:41:33.670602 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.670561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.748108 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.747916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.748108 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.747977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:33.748108 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.748071    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.748464 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.748196    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.748172083 +0000 UTC m=+71.087565665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:33.748464 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.748232    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:33.748464 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.748310    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.748291466 +0000 UTC m=+71.087685051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:33.848942 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.848900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:33.849149 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.849110    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:41:33.849209 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:33.849192    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert podName:44df5c8b-3917-43fa-b640-104adae7fec0 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:41.849173529 +0000 UTC m=+78.188567098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fb6ww" (UID: "44df5c8b-3917-43fa-b640-104adae7fec0") : secret "networking-console-plugin-cert" not found
Sep 17 17:41:33.899941 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.899891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.999339 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:33.999239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:34.511072 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:34.511043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:41:34.511513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:34.511465    2580 scope.go:117] "RemoveContainer" containerID="6fc2f027ad2e6d4fe638bbddec425d77dcb8e2fa2f8cdafa087c440a2c794de3"
Sep 17 17:41:34.511639 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:34.511621    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jnfbk_openshift-console-operator(feae8d3b-b661-4b65-8713-fbf8404a86ed)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" podUID="feae8d3b-b661-4b65-8713-fbf8404a86ed"
Sep 17 17:41:34.685210 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:34.685182    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f2lj4_23fef4bc-4131-4d25-b641-686631df7b5e/dns-node-resolver/0.log"
Sep 17 17:41:34.758047 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:34.758001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:34.758047 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:34.758053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:34.758347 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:34.758203    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:34.758347 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:34.758262    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:34.758347 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:34.758283    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.758263348 +0000 UTC m=+73.097656940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:34.758347 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:34.758333    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.758314611 +0000 UTC m=+73.097708205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:35.066196 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:35.066153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:35.136409 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:35.136371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:35.489233 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:35.489203    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hxlp_3f49d3fa-4582-4508-8858-4596395b5f57/node-ca/0.log"
Sep 17 17:41:36.080688 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.080628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:36.084995 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.084953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j58hh_bd00e048-2992-4249-9ffd-d4d56bb6b446/serve-healthcheck-canary/0.log"
Sep 17 17:41:36.166559 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.166512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:36.167158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.167138    2580 scope.go:117] "RemoveContainer" containerID="6fc2f027ad2e6d4fe638bbddec425d77dcb8e2fa2f8cdafa087c440a2c794de3"
Sep 17 17:41:36.167421 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:36.167387    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jnfbk_openshift-console-operator(feae8d3b-b661-4b65-8713-fbf8404a86ed)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" podUID="feae8d3b-b661-4b65-8713-fbf8404a86ed"
Sep 17 17:41:36.519103 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.519046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tnhn" event={"ID":"45cd1091-7a05-4050-9ea5-fc2c920202bf","Type":"ContainerStarted","Data":"1551e37a219227998c183eb044478d5bf80b1fa1c4985b6e27e3f9aea068a4aa"}
Sep 17 17:41:36.536517 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.536456    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2tnhn" podStartSLOduration=68.148585223 podStartE2EDuration="1m12.53643816s" totalImagesPullingTime="4.387852937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:36.535287517 +0000 UTC m=+72.874681119" watchObservedRunningTime="2026-09-17 17:41:36.53643816 +0000 UTC m=+72.875831750"
Sep 17 17:41:36.780432 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.780317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:36.780432 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:36.780368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:36.780657 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:36.780445    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:36.780657 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:36.780539    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:36.780657 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:36.780547    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.780527182 +0000 UTC m=+77.119920754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:36.780775 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:36.780658    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.780639379 +0000 UTC m=+77.120032952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:37.023518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:37.023467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:37.125829 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:37.125694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:39.440983 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:39.440943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:39.796300 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:39.796175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:39.951122 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:39.951064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.021271 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:40.021231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.123110 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:40.122970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.223184 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:40.223131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.323788 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:40.323745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.695358 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:40.695321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.819727 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:40.819680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:40.819727 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:40.819726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:40.819995 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:40.819813    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:40.819995 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:40.819884    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:40.819995 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:40.819886    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:48.819868728 +0000 UTC m=+85.159262300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:40.819995 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:40.819948    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:48.819931426 +0000 UTC m=+85.159325008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:41.301072 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.301034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.485285 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.485246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:41.485677 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.485663    2580 scope.go:117] "RemoveContainer" containerID="6fc2f027ad2e6d4fe638bbddec425d77dcb8e2fa2f8cdafa087c440a2c794de3"
Sep 17 17:41:41.485836 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.485818    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jnfbk_openshift-console-operator(feae8d3b-b661-4b65-8713-fbf8404a86ed)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" podUID="feae8d3b-b661-4b65-8713-fbf8404a86ed"
Sep 17 17:41:41.627654 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.627519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:41.627654 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.627610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:41.627925 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.627835    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:41.627925 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.627920    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls podName:a0c85c0e-7126-471d-bd1f-292693933089 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:57.627900869 +0000 UTC m=+93.967294445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xqcxc" (UID: "a0c85c0e-7126-471d-bd1f-292693933089") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:41:41.630107 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.630066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/19dd41c1-e0dc-4dee-962f-ee14a11d0438-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8n6t\" (UID: \"19dd41c1-e0dc-4dee-962f-ee14a11d0438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:41.709502 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.709460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"
Sep 17 17:41:41.729127 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.729079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:41.729273 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.729195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:41.729273 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.729217    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:57.729196207 +0000 UTC m=+94.068589775 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:41:41.729393 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.729278    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:41:41.729393 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.729329    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs podName:63f2f4e6-aba2-4b69-8ef9-f2b68013d13e nodeName:}" failed. No retries permitted until 2026-09-17 17:41:57.729317042 +0000 UTC m=+94.068710610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs") pod "router-default-79d7b547b8-gfldz" (UID: "63f2f4e6-aba2-4b69-8ef9-f2b68013d13e") : secret "router-metrics-certs-default" not found
Sep 17 17:41:41.889850 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.889758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.931459 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.931422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:41.931632 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.931583    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:41:41.931682 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:41.931664    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert podName:44df5c8b-3917-43fa-b640-104adae7fec0 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:57.931647932 +0000 UTC m=+94.271041506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fb6ww" (UID: "44df5c8b-3917-43fa-b640-104adae7fec0") : secret "networking-console-plugin-cert" not found
Sep 17 17:41:41.985417 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:41.985382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t"]
Sep 17 17:41:42.090810 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:42.090762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.494952 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:42.494908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.541907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:42.541869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t" event={"ID":"19dd41c1-e0dc-4dee-962f-ee14a11d0438","Type":"ContainerStarted","Data":"b52b70245b28e502040549c546d528dba9c149c2d24a99df2f723a269c156a72"}
Sep 17 17:41:42.696691 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:42.696583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.137582 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:43.137523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.496994 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:43.496960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.702556 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:43.702504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.803565 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:43.803524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:44.090352 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:44.090253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:44.293150 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:44.293116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:44.393998 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:44.393953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:44.550510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:44.550477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t" event={"ID":"19dd41c1-e0dc-4dee-962f-ee14a11d0438","Type":"ContainerStarted","Data":"461674064db15737cd7850bc968f95ef118eed5875017c850027bb7519e568d9"}
Sep 17 17:41:44.550732 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:44.550535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t" event={"ID":"19dd41c1-e0dc-4dee-962f-ee14a11d0438","Type":"ContainerStarted","Data":"20f1efc55ba5e11821e469fd805360b7cfc2b70e3e3dcb8ff08ba9a546a409e0"}
Sep 17 17:41:44.572614 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:44.572561    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8n6t" podStartSLOduration=17.93276748 podStartE2EDuration="19.57254174s" totalImagesPullingTime="1.63977426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:44.570527343 +0000 UTC m=+80.909920934" watchObservedRunningTime="2026-09-17 17:41:44.57254174 +0000 UTC m=+80.911935331"
Sep 17 17:41:44.708427 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:44.708321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.503569 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:45.503518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.691383 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:45.691342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.061753 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:46.061712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.165298 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:46.165254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.264015 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:46.263979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.364695 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:46.364587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.636271 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:46.636179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.736315 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:46.736278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.332144 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:47.332105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.695614 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:47.695577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.911196 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:47.911159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.294133 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:48.294073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.893004 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:48.892967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:48.893004 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:48.893004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:48.893477 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:48.893138    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:48.893477 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:41:48.893202    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls podName:a2f72097-38a0-4d9c-bace-cffca394ce07 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.893183833 +0000 UTC m=+101.232577424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7dtsh" (UID: "a2f72097-38a0-4d9c-bace-cffca394ce07") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:48.893578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:48.893559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2f72097-38a0-4d9c-bace-cffca394ce07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:41:49.127302 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:49.127257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.336349 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:49.336251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.510728 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:49.510687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.696851 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:49.696790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:50.297611 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:50.297574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:50.508664 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:50.508628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.903133 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.903097    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"]
Sep 17 17:41:55.908104 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.908059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:55.913659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.913628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:41:55.913813 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.913628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:41:55.926395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.926365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:41:55.926626 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.926602    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:41:55.928266 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.928242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"]
Sep 17 17:41:55.954973 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.954933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fa5cb720-005b-48e7-8295-f1a8b069116b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:55.955174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.954994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfcm\" (UniqueName: \"kubernetes.io/projected/fa5cb720-005b-48e7-8295-f1a8b069116b-kube-api-access-jtfcm\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:55.955174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.955127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa5cb720-005b-48e7-8295-f1a8b069116b-tmp\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:55.993812 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:55.993766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:56.055833 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.055791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fa5cb720-005b-48e7-8295-f1a8b069116b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:56.056018 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.055870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfcm\" (UniqueName: \"kubernetes.io/projected/fa5cb720-005b-48e7-8295-f1a8b069116b-kube-api-access-jtfcm\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:56.056018 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.055965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa5cb720-005b-48e7-8295-f1a8b069116b-tmp\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:56.056416 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.056388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa5cb720-005b-48e7-8295-f1a8b069116b-tmp\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:56.058632 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.058613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fa5cb720-005b-48e7-8295-f1a8b069116b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:56.070871 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.070839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfcm\" (UniqueName: \"kubernetes.io/projected/fa5cb720-005b-48e7-8295-f1a8b069116b-kube-api-access-jtfcm\") pod \"klusterlet-addon-workmgr-5c8dbb8748-tvf46\" (UID: \"fa5cb720-005b-48e7-8295-f1a8b069116b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:56.218337 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.218239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:41:56.229646 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.229605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:56.333987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.333943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:56.497164 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.497071    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"]
Sep 17 17:41:56.499345 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:56.499312    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5cb720_005b_48e7_8295_f1a8b069116b.slice/crio-71da8ee91f2c9897e167f0b87cefad8e1e74ae94c6cf43419058440f68c03184 WatchSource:0}: Error finding container 71da8ee91f2c9897e167f0b87cefad8e1e74ae94c6cf43419058440f68c03184: Status 404 returned error can't find the container with id 71da8ee91f2c9897e167f0b87cefad8e1e74ae94c6cf43419058440f68c03184
Sep 17 17:41:56.589909 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:56.589867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46" event={"ID":"fa5cb720-005b-48e7-8295-f1a8b069116b","Type":"ContainerStarted","Data":"71da8ee91f2c9897e167f0b87cefad8e1e74ae94c6cf43419058440f68c03184"}
Sep 17 17:41:57.040790 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.040754    2580 scope.go:117] "RemoveContainer" containerID="6fc2f027ad2e6d4fe638bbddec425d77dcb8e2fa2f8cdafa087c440a2c794de3"
Sep 17 17:41:57.606459 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.606419    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:41:57.606713 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.606528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk" event={"ID":"feae8d3b-b661-4b65-8713-fbf8404a86ed","Type":"ContainerStarted","Data":"6d09726e5fe310096d83e5ed0e52d1618d730a8fcf7746ebc12c6784d89ce864"}
Sep 17 17:41:57.606967 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.606941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:57.614122 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.614061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-jnfbk"
Sep 17 17:41:57.674114 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.673674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:57.677349 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.677317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0c85c0e-7126-471d-bd1f-292693933089-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xqcxc\" (UID: \"a0c85c0e-7126-471d-bd1f-292693933089\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:57.774412 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.774345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:57.774597 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.774490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:57.775214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.775188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-service-ca-bundle\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:57.779295 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.779265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2f4e6-aba2-4b69-8ef9-f2b68013d13e-metrics-certs\") pod \"router-default-79d7b547b8-gfldz\" (UID: \"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e\") " pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:57.917324 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.917269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"
Sep 17 17:41:57.927765 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.927735    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-86lrx"]
Sep 17 17:41:57.931218 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.931192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-86lrx"
Sep 17 17:41:57.940158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.940133    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:41:57.940675 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.940649    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:41:57.940811 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.940766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-77zcl\""
Sep 17 17:41:57.970499 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.970457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.975553 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.975499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb594\" (UniqueName: \"kubernetes.io/projected/cca89106-6d22-480b-a8b9-b76c470fd166-kube-api-access-wb594\") pod \"downloads-6594f4c75-86lrx\" (UID: \"cca89106-6d22-480b-a8b9-b76c470fd166\") " pod="openshift-console/downloads-6594f4c75-86lrx"
Sep 17 17:41:57.975738 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.975681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:57.976868 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.976843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-86lrx"]
Sep 17 17:41:57.978844 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.978822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44df5c8b-3917-43fa-b640-104adae7fec0-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fb6ww\" (UID: \"44df5c8b-3917-43fa-b640-104adae7fec0\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:57.986805 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:57.986755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:58.077208 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.077165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb594\" (UniqueName: \"kubernetes.io/projected/cca89106-6d22-480b-a8b9-b76c470fd166-kube-api-access-wb594\") pod \"downloads-6594f4c75-86lrx\" (UID: \"cca89106-6d22-480b-a8b9-b76c470fd166\") " pod="openshift-console/downloads-6594f4c75-86lrx"
Sep 17 17:41:58.098335 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.098302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb594\" (UniqueName: \"kubernetes.io/projected/cca89106-6d22-480b-a8b9-b76c470fd166-kube-api-access-wb594\") pod \"downloads-6594f4c75-86lrx\" (UID: \"cca89106-6d22-480b-a8b9-b76c470fd166\") " pod="openshift-console/downloads-6594f4c75-86lrx"
Sep 17 17:41:58.145395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.145348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.165269 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.165225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"
Sep 17 17:41:58.227320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.227269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.243177 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.242856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-86lrx"
Sep 17 17:41:58.304518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.304466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.455910 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.455783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.580134 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.580108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.781567 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.781475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.824308 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.824269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww"]
Sep 17 17:41:58.829361 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.829330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-86lrx"]
Sep 17 17:41:58.829637 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:58.829571    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44df5c8b_3917_43fa_b640_104adae7fec0.slice/crio-1b034d9bbd8f1b7e14c13fef722675170ddee7bd0f2333ee6986b2796983c45d WatchSource:0}: Error finding container 1b034d9bbd8f1b7e14c13fef722675170ddee7bd0f2333ee6986b2796983c45d: Status 404 returned error can't find the container with id 1b034d9bbd8f1b7e14c13fef722675170ddee7bd0f2333ee6986b2796983c45d
Sep 17 17:41:58.833062 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:58.832990    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca89106_6d22_480b_a8b9_b76c470fd166.slice/crio-1367d06d2831e806cf5fdbf2c58b9d8da1041fd69b98cfab9ec52d36250daaec WatchSource:0}: Error finding container 1367d06d2831e806cf5fdbf2c58b9d8da1041fd69b98cfab9ec52d36250daaec: Status 404 returned error can't find the container with id 1367d06d2831e806cf5fdbf2c58b9d8da1041fd69b98cfab9ec52d36250daaec
Sep 17 17:41:58.834399 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.834372    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc"]
Sep 17 17:41:58.836214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.836061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79d7b547b8-gfldz"]
Sep 17 17:41:58.836214 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:58.836180    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c85c0e_7126_471d_bd1f_292693933089.slice/crio-d7170c5598871b44bc6f39d1bfcf518a38ccc19989fdf529ac90408202bbbe26 WatchSource:0}: Error finding container d7170c5598871b44bc6f39d1bfcf518a38ccc19989fdf529ac90408202bbbe26: Status 404 returned error can't find the container with id d7170c5598871b44bc6f39d1bfcf518a38ccc19989fdf529ac90408202bbbe26
Sep 17 17:41:58.839618 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:41:58.839590    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f2f4e6_aba2_4b69_8ef9_f2b68013d13e.slice/crio-5be0286c61de02ba6dd4f7040d27b268d22ff6bf24f1f41a19a534420f80e928 WatchSource:0}: Error finding container 5be0286c61de02ba6dd4f7040d27b268d22ff6bf24f1f41a19a534420f80e928: Status 404 returned error can't find the container with id 5be0286c61de02ba6dd4f7040d27b268d22ff6bf24f1f41a19a534420f80e928
Sep 17 17:41:58.878402 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:58.878356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.091206 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.087945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.158824 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.158780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.620258 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.620130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww" event={"ID":"44df5c8b-3917-43fa-b640-104adae7fec0","Type":"ContainerStarted","Data":"1b034d9bbd8f1b7e14c13fef722675170ddee7bd0f2333ee6986b2796983c45d"}
Sep 17 17:41:59.622468 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.622380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc" event={"ID":"a0c85c0e-7126-471d-bd1f-292693933089","Type":"ContainerStarted","Data":"d7170c5598871b44bc6f39d1bfcf518a38ccc19989fdf529ac90408202bbbe26"}
Sep 17 17:41:59.625969 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.625928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79d7b547b8-gfldz" event={"ID":"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e","Type":"ContainerStarted","Data":"305fcc4254ab018196ac559e1a67a77d7d1f9d46a165666030123c3fa182b7b0"}
Sep 17 17:41:59.626160 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.625996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79d7b547b8-gfldz" event={"ID":"63f2f4e6-aba2-4b69-8ef9-f2b68013d13e","Type":"ContainerStarted","Data":"5be0286c61de02ba6dd4f7040d27b268d22ff6bf24f1f41a19a534420f80e928"}
Sep 17 17:41:59.628719 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.628261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-86lrx" event={"ID":"cca89106-6d22-480b-a8b9-b76c470fd166","Type":"ContainerStarted","Data":"1367d06d2831e806cf5fdbf2c58b9d8da1041fd69b98cfab9ec52d36250daaec"}
Sep 17 17:41:59.650179 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.649709    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-79d7b547b8-gfldz" podStartSLOduration=34.649683628 podStartE2EDuration="34.649683628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:59.648708715 +0000 UTC m=+95.988102307" watchObservedRunningTime="2026-09-17 17:41:59.649683628 +0000 UTC m=+95.989077220"
Sep 17 17:41:59.673445 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.673383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.859287 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.858950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.987178 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.987140    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:41:59.991163 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:41:59.991130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:42:00.138957 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.138882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:00.377575 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.377446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:00.494430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.494365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:00.634990 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.634885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww" event={"ID":"44df5c8b-3917-43fa-b640-104adae7fec0","Type":"ContainerStarted","Data":"c11d143a3144929b5e2837ce97d22ead4f56021726a0aaf02103df8086d03e83"}
Sep 17 17:42:00.635480 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.635389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:42:00.636907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.636880    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79d7b547b8-gfldz"
Sep 17 17:42:00.666971 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.666904    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fb6ww" podStartSLOduration=33.413716303 podStartE2EDuration="34.666877045s" totalImagesPullingTime="1.253160742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:00.664631082 +0000 UTC m=+97.004024686" watchObservedRunningTime="2026-09-17 17:42:00.666877045 +0000 UTC m=+97.006270637"
Sep 17 17:42:00.884928 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.884860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:00.977063 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:00.976926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.198791 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.198725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.300111 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.300008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.385865 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.385806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.391398 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.390849    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"]
Sep 17 17:42:01.394680 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.394514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"
Sep 17 17:42:01.397282 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.397253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nz4md\""
Sep 17 17:42:01.397604 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.397582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:42:01.407562 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.407538    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"]
Sep 17 17:42:01.512018 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.511755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21bd49d9-5444-4a12-9481-885bfad0e3e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-gv9n6\" (UID: \"21bd49d9-5444-4a12-9481-885bfad0e3e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"
Sep 17 17:42:01.563222 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.563120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.613152 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.613113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21bd49d9-5444-4a12-9481-885bfad0e3e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-gv9n6\" (UID: \"21bd49d9-5444-4a12-9481-885bfad0e3e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"
Sep 17 17:42:01.616767 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.616730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21bd49d9-5444-4a12-9481-885bfad0e3e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-gv9n6\" (UID: \"21bd49d9-5444-4a12-9481-885bfad0e3e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"
Sep 17 17:42:01.646600 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.646527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc" event={"ID":"a0c85c0e-7126-471d-bd1f-292693933089","Type":"ContainerStarted","Data":"9b2be1ba194117b69603537fbb06466db46cfa7c98fd547a3b4f3def08fd29d5"}
Sep 17 17:42:01.667684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.667610    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xqcxc" podStartSLOduration=34.698994237 podStartE2EDuration="36.667583959s" totalImagesPullingTime="1.968589722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.666654308 +0000 UTC m=+98.006047898" watchObservedRunningTime="2026-09-17 17:42:01.667583959 +0000 UTC m=+98.006977553"
Sep 17 17:42:01.708027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:01.707997    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"
Sep 17 17:42:02.013841 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.013805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"]
Sep 17 17:42:02.017248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.017199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:42:02.017378 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.017312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:42:02.021071 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.021037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a97cf2-225e-4096-9b32-05fe7c4f9e75-metrics-tls\") pod \"dns-default-blzqz\" (UID: \"24a97cf2-225e-4096-9b32-05fe7c4f9e75\") " pod="openshift-dns/dns-default-blzqz"
Sep 17 17:42:02.021283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.021259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"image-registry-5c4c4f9c74-szdr7\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") " pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:42:02.284105 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.284057    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mkt75\""
Sep 17 17:42:02.291503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.291463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:42:02.305126 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.305074    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n5jl\""
Sep 17 17:42:02.310101 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.310055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-blzqz"
Sep 17 17:42:02.358601 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:02.358446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:03.224332 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:42:03.224285    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bd49d9_5444_4a12_9481_885bfad0e3e7.slice/crio-7b42a4111a40fa00b0c2e2ac04d55a2de5bc7b3ee4cbd47ab4b8b4d2a401f50d WatchSource:0}: Error finding container 7b42a4111a40fa00b0c2e2ac04d55a2de5bc7b3ee4cbd47ab4b8b4d2a401f50d: Status 404 returned error can't find the container with id 7b42a4111a40fa00b0c2e2ac04d55a2de5bc7b3ee4cbd47ab4b8b4d2a401f50d
Sep 17 17:42:03.513859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:03.513759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-blzqz"]
Sep 17 17:42:03.517951 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:03.517926    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c4c4f9c74-szdr7"]
Sep 17 17:42:03.650533 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:42:03.650494    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a97cf2_225e_4096_9b32_05fe7c4f9e75.slice/crio-0d2e98dcad2de3d1704404d761bbfe0109fe61e23543bbefecf8749945d8b794 WatchSource:0}: Error finding container 0d2e98dcad2de3d1704404d761bbfe0109fe61e23543bbefecf8749945d8b794: Status 404 returned error can't find the container with id 0d2e98dcad2de3d1704404d761bbfe0109fe61e23543bbefecf8749945d8b794
Sep 17 17:42:03.651647 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:42:03.651600    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1081e7e3_d38e_4fa8_827a_85928fbd4970.slice/crio-42bbf3180db42d0c9533f48dc587351d7f0e0b409f1d4cab5364d5f09ae4d5b3 WatchSource:0}: Error finding container 42bbf3180db42d0c9533f48dc587351d7f0e0b409f1d4cab5364d5f09ae4d5b3: Status 404 returned error can't find the container with id 42bbf3180db42d0c9533f48dc587351d7f0e0b409f1d4cab5364d5f09ae4d5b3
Sep 17 17:42:03.659126 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:03.659064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7" event={"ID":"1081e7e3-d38e-4fa8-827a-85928fbd4970","Type":"ContainerStarted","Data":"42bbf3180db42d0c9533f48dc587351d7f0e0b409f1d4cab5364d5f09ae4d5b3"}
Sep 17 17:42:03.660608 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:03.660564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6" event={"ID":"21bd49d9-5444-4a12-9481-885bfad0e3e7","Type":"ContainerStarted","Data":"7b42a4111a40fa00b0c2e2ac04d55a2de5bc7b3ee4cbd47ab4b8b4d2a401f50d"}
Sep 17 17:42:03.662031 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:03.661995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blzqz" event={"ID":"24a97cf2-225e-4096-9b32-05fe7c4f9e75","Type":"ContainerStarted","Data":"0d2e98dcad2de3d1704404d761bbfe0109fe61e23543bbefecf8749945d8b794"}
Sep 17 17:42:04.515932 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.515272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z689h"
Sep 17 17:42:04.669118 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.669056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7" event={"ID":"1081e7e3-d38e-4fa8-827a-85928fbd4970","Type":"ContainerStarted","Data":"29a6dec364689a151125b97225f1142615d70f73c9032d94a95f7d5d727f3013"}
Sep 17 17:42:04.669340 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.669320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:42:04.671336 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.671301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46" event={"ID":"fa5cb720-005b-48e7-8295-f1a8b069116b","Type":"ContainerStarted","Data":"84c70f6432bcdd05257efbd7d8cb522a12ee7fd600eae28e5456347e03d88fb7"}
Sep 17 17:42:04.671918 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.671872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:42:04.674133 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.674061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46"
Sep 17 17:42:04.719847 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.719776    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7" podStartSLOduration=100.719733003 podStartE2EDuration="1m40.719733003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:04.719195803 +0000 UTC m=+101.058589395" watchObservedRunningTime="2026-09-17 17:42:04.719733003 +0000 UTC m=+101.059126592"
Sep 17 17:42:04.949534 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.949484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:42:04.952532 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:04.952503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2f72097-38a0-4d9c-bace-cffca394ce07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dtsh\" (UID: \"a2f72097-38a0-4d9c-bace-cffca394ce07\") " pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:42:05.115955 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:05.115894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7dtsh"
Sep 17 17:42:05.364468 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:05.364332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.471519 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:05.471478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.887047 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:05.886955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.900611 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:05.900548    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8dbb8748-tvf46" podStartSLOduration=3.710470704 podStartE2EDuration="10.90052605s" totalImagesPullingTime="7.190055346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:04.764182444 +0000 UTC m=+101.103576037" watchObservedRunningTime="2026-09-17 17:42:05.90052605 +0000 UTC m=+102.239919641"
Sep 17 17:42:05.900955 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:05.900921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7dtsh"]
Sep 17 17:42:05.961263 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:42:05.961215    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f72097_38a0_4d9c_bace_cffca394ce07.slice/crio-307b75aff36b6352dec0036f19c88543579efd2b66a7dc9fb4ba0b715114c547 WatchSource:0}: Error finding container 307b75aff36b6352dec0036f19c88543579efd2b66a7dc9fb4ba0b715114c547: Status 404 returned error can't find the container with id 307b75aff36b6352dec0036f19c88543579efd2b66a7dc9fb4ba0b715114c547
Sep 17 17:42:05.991597 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:05.991425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:06.684840 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.684795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6" event={"ID":"21bd49d9-5444-4a12-9481-885bfad0e3e7","Type":"ContainerStarted","Data":"7aa6b8549ab3fa985c3b97c0bb2295ec1d3a9457c8947ee80be9831a4f919c52"}
Sep 17 17:42:06.685048 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.684876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"
Sep 17 17:42:06.687991 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.687960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blzqz" event={"ID":"24a97cf2-225e-4096-9b32-05fe7c4f9e75","Type":"ContainerStarted","Data":"2a2d8722706703eafa310755f70438d8dee41a6425611e1c387677ba20c3d6eb"}
Sep 17 17:42:06.688168 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.688013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blzqz" event={"ID":"24a97cf2-225e-4096-9b32-05fe7c4f9e75","Type":"ContainerStarted","Data":"76cbd533b08740124e38ece1260dfcdf138e6c8256bad95ff4c3a1529fbe5dec"}
Sep 17 17:42:06.688168 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.688077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-blzqz"
Sep 17 17:42:06.690367 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.690334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dtsh" event={"ID":"a2f72097-38a0-4d9c-bace-cffca394ce07","Type":"ContainerStarted","Data":"ec58c4ef9008b4038f24310d8b05a81ed6d105c9940dd233d63e1cb7cea02bc2"}
Sep 17 17:42:06.690512 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.690397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dtsh" event={"ID":"a2f72097-38a0-4d9c-bace-cffca394ce07","Type":"ContainerStarted","Data":"307b75aff36b6352dec0036f19c88543579efd2b66a7dc9fb4ba0b715114c547"}
Sep 17 17:42:06.691710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.691687    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6"
Sep 17 17:42:06.703828 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.703755    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-gv9n6" podStartSLOduration=3.310847308 podStartE2EDuration="5.703736559s" totalImagesPullingTime="2.392889251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:06.703189078 +0000 UTC m=+103.042582696" watchObservedRunningTime="2026-09-17 17:42:06.703736559 +0000 UTC m=+103.043130155"
Sep 17 17:42:06.725076 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:06.725020    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-blzqz" podStartSLOduration=66.413846761 podStartE2EDuration="1m8.725000854s" totalImagesPullingTime="2.311154093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:06.724283466 +0000 UTC m=+103.063677062" watchObservedRunningTime="2026-09-17 17:42:06.725000854 +0000 UTC m=+103.064394445"
Sep 17 17:42:07.117967 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:07.117926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.341266 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:07.341126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.483039 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:07.482995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.699080 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:07.699037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dtsh" event={"ID":"a2f72097-38a0-4d9c-bace-cffca394ce07","Type":"ContainerStarted","Data":"fa257458d5348b4dd2e56d9899633c9e3f7b67201b9fea2d3bc3d67de02a9305"}
Sep 17 17:42:07.728443 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:07.728407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:08.050351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:08.050154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:08.176684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:08.176594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:10.853879 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:10.853830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:10.965786 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:10.965749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:11.070708 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:11.070662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:11.734052 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:11.733931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:11.837311 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:11.837257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:12.220926 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:12.220815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:14.725277 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:14.725217    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:14.832952 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:14.831542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:14.948282 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:14.948236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:15.194077 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.194038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:15.307344 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.307303    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q868m"]
Sep 17 17:42:15.314946 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.314909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.320391 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.320361    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qffnd\""
Sep 17 17:42:15.320567 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.320361    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:42:15.320656 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.320594    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:42:15.320656 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.320612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:42:15.321549 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.321531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:42:15.436350 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.436310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:15.454478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgb2t\" (UniqueName: \"kubernetes.io/projected/a101289d-9b18-456f-84d7-86a34dab5d74-kube-api-access-xgb2t\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.454478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-sys\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.454478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-wtmp\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.454758 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-accelerators-collector-config\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.454758 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-root\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.454758 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.454758 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-textfile\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.454758 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-tls\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.455005 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.454792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a101289d-9b18-456f-84d7-86a34dab5d74-metrics-client-ca\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.531191 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.531132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:15.556285 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a101289d-9b18-456f-84d7-86a34dab5d74-metrics-client-ca\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgb2t\" (UniqueName: \"kubernetes.io/projected/a101289d-9b18-456f-84d7-86a34dab5d74-kube-api-access-xgb2t\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-sys\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-wtmp\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-accelerators-collector-config\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-root\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-sys\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556860 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-textfile\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556860 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-tls\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556860 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-wtmp\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556860 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a101289d-9b18-456f-84d7-86a34dab5d74-root\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.556860 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-textfile\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.557132 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.556970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a101289d-9b18-456f-84d7-86a34dab5d74-metrics-client-ca\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.557194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.557178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-accelerators-collector-config\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.559422 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.559391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.559548 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.559532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a101289d-9b18-456f-84d7-86a34dab5d74-node-exporter-tls\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.568587 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.568554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgb2t\" (UniqueName: \"kubernetes.io/projected/a101289d-9b18-456f-84d7-86a34dab5d74-kube-api-access-xgb2t\") pod \"node-exporter-q868m\" (UID: \"a101289d-9b18-456f-84d7-86a34dab5d74\") " pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.628445 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.628402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q868m"
Sep 17 17:42:15.891249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:15.891212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.002721 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.002681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.100065 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.100029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.107062 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:42:16.107024    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda101289d_9b18_456f_84d7_86a34dab5d74.slice/crio-3e03d5eaaa9e38d47ec5d0264390a26e7b0e6f5a98d11643776418c778d3bdaa WatchSource:0}: Error finding container 3e03d5eaaa9e38d47ec5d0264390a26e7b0e6f5a98d11643776418c778d3bdaa: Status 404 returned error can't find the container with id 3e03d5eaaa9e38d47ec5d0264390a26e7b0e6f5a98d11643776418c778d3bdaa
Sep 17 17:42:16.217438 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.217405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.418551 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.418442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.523049 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.522790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.702612 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.702518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-blzqz"
Sep 17 17:42:16.749426 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.746670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-86lrx" event={"ID":"cca89106-6d22-480b-a8b9-b76c470fd166","Type":"ContainerStarted","Data":"1e6e2d2f6574470c9ef221d2fc04d183e7ee8fb0aff84f9a0be9f239e5edbd1d"}
Sep 17 17:42:16.749426 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.747546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-86lrx"
Sep 17 17:42:16.751774 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.751573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dtsh" event={"ID":"a2f72097-38a0-4d9c-bace-cffca394ce07","Type":"ContainerStarted","Data":"f4fc403598c116e94f76c3b89edeb6b022570e51b9084c1252f4f347e6a8dd24"}
Sep 17 17:42:16.753197 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.753103    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-86lrx"
Sep 17 17:42:16.754392 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.754289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q868m" event={"ID":"a101289d-9b18-456f-84d7-86a34dab5d74","Type":"ContainerStarted","Data":"3e03d5eaaa9e38d47ec5d0264390a26e7b0e6f5a98d11643776418c778d3bdaa"}
Sep 17 17:42:16.809762 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.809676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-86lrx" podStartSLOduration=2.446403326 podStartE2EDuration="19.809647437s" totalImagesPullingTime="17.363244111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:16.773462148 +0000 UTC m=+113.112855742" watchObservedRunningTime="2026-09-17 17:42:16.809647437 +0000 UTC m=+113.149041032"
Sep 17 17:42:16.810774 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:16.810722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7dtsh" podStartSLOduration=34.976525479 podStartE2EDuration="44.810704896s" totalImagesPullingTime="9.834179417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:16.799840157 +0000 UTC m=+113.139233761" watchObservedRunningTime="2026-09-17 17:42:16.810704896 +0000 UTC m=+113.150098488"
Sep 17 17:42:17.763493 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:17.763448    2580 generic.go:356] "Generic (PLEG): container finished" podID="a101289d-9b18-456f-84d7-86a34dab5d74" containerID="b6efdf6bdc4f2552e0fe69c35a4000758541a65b39ed12e2540f59f8c9239136" exitCode=0
Sep 17 17:42:17.764032 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:17.763532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q868m" event={"ID":"a101289d-9b18-456f-84d7-86a34dab5d74","Type":"ContainerDied","Data":"b6efdf6bdc4f2552e0fe69c35a4000758541a65b39ed12e2540f59f8c9239136"}
Sep 17 17:42:18.009588 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.009551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.140525 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.140474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.336273 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.336234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.440978 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.440884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.659983 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.659940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.729832 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.729746    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c4c4f9c74-szdr7"]
Sep 17 17:42:18.735012 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.734983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:42:18.774063 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.773920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.774063 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.773996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q868m" event={"ID":"a101289d-9b18-456f-84d7-86a34dab5d74","Type":"ContainerStarted","Data":"e46fee90100c370f839066fbe73842da3fd2e7c67e73121e07d8cba24705f826"}
Sep 17 17:42:18.774632 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.774118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q868m" event={"ID":"a101289d-9b18-456f-84d7-86a34dab5d74","Type":"ContainerStarted","Data":"dc2fa0bbd71ded7ec0f93f292759011d2ac00f27c41a98a040c9d55624b957e7"}
Sep 17 17:42:18.817257 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.817193    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q868m" podStartSLOduration=2.609927047 podStartE2EDuration="3.817169113s" totalImagesPullingTime="1.121033267s" totalInitContainerRuntime="86.208799ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:18.81678911 +0000 UTC m=+115.156182704" watchObservedRunningTime="2026-09-17 17:42:18.817169113 +0000 UTC m=+115.156562705"
Sep 17 17:42:18.894884 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.894830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.995003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:18.994874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:19.767874 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:19.767831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:19.849474 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:19.849431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.148673 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:20.148634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.245495 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:20.245454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.747630 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:20.747590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.832993 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:20.832947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:21.705609 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:21.705569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.969801 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:22.969756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:23.148072 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:23.148032    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:23.987113 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:23.987060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:24.064221 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:24.064152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.510307 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:27.510247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.610647 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:27.610610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.711908 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:27.711874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:30.157372 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:30.157324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:30.257935 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:30.257894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:30.357790 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:30.357752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:33.042820 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:33.042783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:33.143702 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:33.143665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:33.927710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:33.927658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:42:33.930158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:33.930133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c77eba-b3f9-4472-8dad-5b68b500e7dd-metrics-certs\") pod \"network-metrics-daemon-d67rh\" (UID: \"78c77eba-b3f9-4472-8dad-5b68b500e7dd\") " pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:42:34.117259 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:34.117227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7j64\""
Sep 17 17:42:34.125415 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:34.125388    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d67rh"
Sep 17 17:42:34.449270 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:34.449242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d67rh"]
Sep 17 17:42:34.452012 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:42:34.451980    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c77eba_b3f9_4472_8dad_5b68b500e7dd.slice/crio-406be777cee2589a39b0b488b3865bb2d10d66da847c4183298bb1a05e37d806 WatchSource:0}: Error finding container 406be777cee2589a39b0b488b3865bb2d10d66da847c4183298bb1a05e37d806: Status 404 returned error can't find the container with id 406be777cee2589a39b0b488b3865bb2d10d66da847c4183298bb1a05e37d806
Sep 17 17:42:34.851812 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:34.851733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d67rh" event={"ID":"78c77eba-b3f9-4472-8dad-5b68b500e7dd","Type":"ContainerStarted","Data":"406be777cee2589a39b0b488b3865bb2d10d66da847c4183298bb1a05e37d806"}
Sep 17 17:42:36.866508 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:36.866468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d67rh" event={"ID":"78c77eba-b3f9-4472-8dad-5b68b500e7dd","Type":"ContainerStarted","Data":"024a925811019c0d9c559306e6f5cf73880612ff5261703b0a3931fadfd24cd7"}
Sep 17 17:42:36.867068 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:36.866570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d67rh" event={"ID":"78c77eba-b3f9-4472-8dad-5b68b500e7dd","Type":"ContainerStarted","Data":"1860536d09c1242fb83b03ab2e36829ca37eb44f300d673a046f54defbecdb03"}
Sep 17 17:42:36.885882 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:36.885827    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d67rh" podStartSLOduration=131.598308023 podStartE2EDuration="2m12.885810689s" totalImagesPullingTime="1.287502666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:36.884992238 +0000 UTC m=+133.224385828" watchObservedRunningTime="2026-09-17 17:42:36.885810689 +0000 UTC m=+133.225204279"
Sep 17 17:42:37.004364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:37.004310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:37.104943 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:37.104909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:37.871697 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:37.871660    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6481e0f-0af5-42c7-86ac-0c268a08bb19" containerID="864f7effab4e9d9d9b893a07eff2c601a2b3fcd8663e38a1d03ae05a0304861e" exitCode=0
Sep 17 17:42:37.872140 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:37.871737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q" event={"ID":"f6481e0f-0af5-42c7-86ac-0c268a08bb19","Type":"ContainerDied","Data":"864f7effab4e9d9d9b893a07eff2c601a2b3fcd8663e38a1d03ae05a0304861e"}
Sep 17 17:42:37.872229 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:37.872210    2580 scope.go:117] "RemoveContainer" containerID="864f7effab4e9d9d9b893a07eff2c601a2b3fcd8663e38a1d03ae05a0304861e"
Sep 17 17:42:38.258698 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:38.258660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:38.321159 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:38.321120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:38.387506 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:38.387466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:38.877425 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:38.877385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dmz2q" event={"ID":"f6481e0f-0af5-42c7-86ac-0c268a08bb19","Type":"ContainerStarted","Data":"3dd3424842bb2802450a16ac92a55704620a60982b1739528ac359351228d396"}
Sep 17 17:42:43.753240 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:43.753169    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7" podUID="1081e7e3-d38e-4fa8-827a-85928fbd4970" containerName="registry" containerID="cri-o://29a6dec364689a151125b97225f1142615d70f73c9032d94a95f7d5d727f3013" gracePeriod=30
Sep 17 17:42:43.900265 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:43.900215    2580 generic.go:356] "Generic (PLEG): container finished" podID="1081e7e3-d38e-4fa8-827a-85928fbd4970" containerID="29a6dec364689a151125b97225f1142615d70f73c9032d94a95f7d5d727f3013" exitCode=0
Sep 17 17:42:43.900463 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:43.900281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7" event={"ID":"1081e7e3-d38e-4fa8-827a-85928fbd4970","Type":"ContainerDied","Data":"29a6dec364689a151125b97225f1142615d70f73c9032d94a95f7d5d727f3013"}
Sep 17 17:42:44.006499 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.006434    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:42:44.121849 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.121814    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.121849 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.121851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25vxq\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-kube-api-access-25vxq\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.122113 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.121870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-bound-sa-token\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.122113 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.121922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-image-registry-private-configuration\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.122113 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.121957    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-installation-pull-secrets\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.122113 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.121996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1081e7e3-d38e-4fa8-827a-85928fbd4970-ca-trust-extracted\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.122328 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.122187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-certificates\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.122328 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.122250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-trusted-ca\") pod \"1081e7e3-d38e-4fa8-827a-85928fbd4970\" (UID: \"1081e7e3-d38e-4fa8-827a-85928fbd4970\") "
Sep 17 17:42:44.122941 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.122843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:44.123061 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.122939    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:44.124858 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.124816    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:44.125073 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.125039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:44.125214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.125099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:44.125214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.125157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:44.125333 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.125200    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-kube-api-access-25vxq" (OuterVolumeSpecName: "kube-api-access-25vxq") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "kube-api-access-25vxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:44.131504 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.131473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1081e7e3-d38e-4fa8-827a-85928fbd4970-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1081e7e3-d38e-4fa8-827a-85928fbd4970" (UID: "1081e7e3-d38e-4fa8-827a-85928fbd4970"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:44.223890 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223828    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-image-registry-private-configuration\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.223890 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223882    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1081e7e3-d38e-4fa8-827a-85928fbd4970-installation-pull-secrets\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.223890 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223892    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1081e7e3-d38e-4fa8-827a-85928fbd4970-ca-trust-extracted\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.223890 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223900    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-certificates\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.223890 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223909    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1081e7e3-d38e-4fa8-827a-85928fbd4970-trusted-ca\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.224253 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223918    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-registry-tls\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.224253 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223926    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25vxq\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-kube-api-access-25vxq\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.224253 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.223934    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1081e7e3-d38e-4fa8-827a-85928fbd4970-bound-sa-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:42:44.905741 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.905716    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7"
Sep 17 17:42:44.906245 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.905712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4c4f9c74-szdr7" event={"ID":"1081e7e3-d38e-4fa8-827a-85928fbd4970","Type":"ContainerDied","Data":"42bbf3180db42d0c9533f48dc587351d7f0e0b409f1d4cab5364d5f09ae4d5b3"}
Sep 17 17:42:44.906245 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.905858    2580 scope.go:117] "RemoveContainer" containerID="29a6dec364689a151125b97225f1142615d70f73c9032d94a95f7d5d727f3013"
Sep 17 17:42:44.929382 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.929352    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c4c4f9c74-szdr7"]
Sep 17 17:42:44.933503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:44.933471    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c4c4f9c74-szdr7"]
Sep 17 17:42:45.287250 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:45.287156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:45.367249 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:45.367212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:46.044640 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:46.044605    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1081e7e3-d38e-4fa8-827a-85928fbd4970" path="/var/lib/kubelet/pods/1081e7e3-d38e-4fa8-827a-85928fbd4970/volumes"
Sep 17 17:42:57.963001 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:57.962967    2580 generic.go:356] "Generic (PLEG): container finished" podID="87cf91a8-0c17-46f0-b87b-b8cce2936c96" containerID="de77ddd565a063c219be116bd44fa87094c1562bd943570a7011fa3eb53bfd7d" exitCode=0
Sep 17 17:42:57.963627 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:57.963040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg" event={"ID":"87cf91a8-0c17-46f0-b87b-b8cce2936c96","Type":"ContainerDied","Data":"de77ddd565a063c219be116bd44fa87094c1562bd943570a7011fa3eb53bfd7d"}
Sep 17 17:42:57.963627 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:57.963413    2580 scope.go:117] "RemoveContainer" containerID="de77ddd565a063c219be116bd44fa87094c1562bd943570a7011fa3eb53bfd7d"
Sep 17 17:42:57.964730 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:57.964707    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6" containerID="5f4a23fea8093f802d576d799aeced2596062f809f40f298a8bb414bf17fb8f5" exitCode=0
Sep 17 17:42:57.964823 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:57.964783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gczxt" event={"ID":"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6","Type":"ContainerDied","Data":"5f4a23fea8093f802d576d799aeced2596062f809f40f298a8bb414bf17fb8f5"}
Sep 17 17:42:57.965061 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:57.965049    2580 scope.go:117] "RemoveContainer" containerID="5f4a23fea8093f802d576d799aeced2596062f809f40f298a8bb414bf17fb8f5"
Sep 17 17:42:58.970442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:58.970401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gs2qg" event={"ID":"87cf91a8-0c17-46f0-b87b-b8cce2936c96","Type":"ContainerStarted","Data":"7c96834685dfaa17b7fa75fcfc16fb3267338aa9fbd67d5497cddfc7654c1397"}
Sep 17 17:42:58.972322 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:42:58.972298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gczxt" event={"ID":"f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6","Type":"ContainerStarted","Data":"952b598707ea1a7bd312df18e249c8e0492a1022fbc565433f706118911871b6"}
Sep 17 17:43:00.303660 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:00.303626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:00.399886 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:00.399849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.688067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:11.688027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:26.842242 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:26.842208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:31.536593 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:31.536557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:31.602354 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:31.602324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:35.486302 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:35.486252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:35.545275 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:35.545239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:43.468048 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:43.468007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:43.534716 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:43.534679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:46.633978 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:46.633942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:46.734205 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:46.734171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:46.795887 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:46.795849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:59.781858 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:59.781816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:59.883107 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:43:59.883052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.295623 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:44:00.295585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.394737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:44:00.394688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:23.354556 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:44:23.354516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:23.456269 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:44:23.456228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:28.685780 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:44:28.685745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:28.757717 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:44:28.757672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:02.111301 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:02.111180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:02.180040 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:02.180002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:02.507327 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:02.507286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:23.952073 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:23.952045    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:45:23.957380 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:23.957348    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:45:23.962902 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:23.962856    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:45:43.312879 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:43.312843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:43.412541 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:43.412499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:43.513311 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:43.513263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:44.967454 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.967386    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"]
Sep 17 17:45:44.967889 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.967798    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1081e7e3-d38e-4fa8-827a-85928fbd4970" containerName="registry"
Sep 17 17:45:44.967889 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.967815    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1081e7e3-d38e-4fa8-827a-85928fbd4970" containerName="registry"
Sep 17 17:45:44.967889 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.967877    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1081e7e3-d38e-4fa8-827a-85928fbd4970" containerName="registry"
Sep 17 17:45:44.971063 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.971018    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:44.974569 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.974534    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ndbbk\""
Sep 17 17:45:44.976167 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.976141    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:45:44.976348 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.976154    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:45:44.982840 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:44.982795    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"]
Sep 17 17:45:45.046745 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.046677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.046745 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.046744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lw6p\" (UniqueName: \"kubernetes.io/projected/34d0542d-1a2f-4d6a-8f62-c35482526b63-kube-api-access-2lw6p\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.047194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.046805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.147914 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.147870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.147914 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.147911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lw6p\" (UniqueName: \"kubernetes.io/projected/34d0542d-1a2f-4d6a-8f62-c35482526b63-kube-api-access-2lw6p\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.148177 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.147960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.148322 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.148301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.148375 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.148355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.160299 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.160274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lw6p\" (UniqueName: \"kubernetes.io/projected/34d0542d-1a2f-4d6a-8f62-c35482526b63-kube-api-access-2lw6p\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.284291 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.284185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:45:45.563922 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.563889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"]
Sep 17 17:45:45.569525 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:45:45.569493    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d0542d_1a2f_4d6a_8f62_c35482526b63.slice/crio-07585607bfb78652f1d3caecf0d7067dd40008329ea09e631d42a6aa0d1b119f WatchSource:0}: Error finding container 07585607bfb78652f1d3caecf0d7067dd40008329ea09e631d42a6aa0d1b119f: Status 404 returned error can't find the container with id 07585607bfb78652f1d3caecf0d7067dd40008329ea09e631d42a6aa0d1b119f
Sep 17 17:45:45.571748 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.571728    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:45:45.611282 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:45.611239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g" event={"ID":"34d0542d-1a2f-4d6a-8f62-c35482526b63","Type":"ContainerStarted","Data":"07585607bfb78652f1d3caecf0d7067dd40008329ea09e631d42a6aa0d1b119f"}
Sep 17 17:45:52.645019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:52.644982    2580 generic.go:356] "Generic (PLEG): container finished" podID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerID="d745bb4ed534428604c6ebdeb22274d6d03f70393cda4349c51dbf14f1d32e1c" exitCode=0
Sep 17 17:45:52.645464 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:52.645067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g" event={"ID":"34d0542d-1a2f-4d6a-8f62-c35482526b63","Type":"ContainerDied","Data":"d745bb4ed534428604c6ebdeb22274d6d03f70393cda4349c51dbf14f1d32e1c"}
Sep 17 17:45:55.660283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:55.660246    2580 generic.go:356] "Generic (PLEG): container finished" podID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerID="3cdbfd033f9154167c6db735657038e1b33ddcc24db3fa9ab4217468abf22864" exitCode=0
Sep 17 17:45:55.660689 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:45:55.660300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g" event={"ID":"34d0542d-1a2f-4d6a-8f62-c35482526b63","Type":"ContainerDied","Data":"3cdbfd033f9154167c6db735657038e1b33ddcc24db3fa9ab4217468abf22864"}
Sep 17 17:46:02.696306 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:02.696262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g" event={"ID":"34d0542d-1a2f-4d6a-8f62-c35482526b63","Type":"ContainerStarted","Data":"fa44ba688165a77db94d4e1484d2e538730352c19b37a9eae7bd44da6dbc5a26"}
Sep 17 17:46:02.717752 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:02.717698    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g" podStartSLOduration=3.585318459 podStartE2EDuration="18.717680073s" totalImagesPullingTime="15.132361614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:02.717310981 +0000 UTC m=+339.056704573" watchObservedRunningTime="2026-09-17 17:46:02.717680073 +0000 UTC m=+339.057073665"
Sep 17 17:46:03.702328 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:03.702287    2580 generic.go:356] "Generic (PLEG): container finished" podID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerID="fa44ba688165a77db94d4e1484d2e538730352c19b37a9eae7bd44da6dbc5a26" exitCode=0
Sep 17 17:46:03.702742 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:03.702370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g" event={"ID":"34d0542d-1a2f-4d6a-8f62-c35482526b63","Type":"ContainerDied","Data":"fa44ba688165a77db94d4e1484d2e538730352c19b37a9eae7bd44da6dbc5a26"}
Sep 17 17:46:04.830753 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:04.830718    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:46:04.925372 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:04.925324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-util\") pod \"34d0542d-1a2f-4d6a-8f62-c35482526b63\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") "
Sep 17 17:46:04.925372 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:04.925370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lw6p\" (UniqueName: \"kubernetes.io/projected/34d0542d-1a2f-4d6a-8f62-c35482526b63-kube-api-access-2lw6p\") pod \"34d0542d-1a2f-4d6a-8f62-c35482526b63\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") "
Sep 17 17:46:04.925597 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:04.925388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-bundle\") pod \"34d0542d-1a2f-4d6a-8f62-c35482526b63\" (UID: \"34d0542d-1a2f-4d6a-8f62-c35482526b63\") "
Sep 17 17:46:04.926260 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:04.926232    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-bundle" (OuterVolumeSpecName: "bundle") pod "34d0542d-1a2f-4d6a-8f62-c35482526b63" (UID: "34d0542d-1a2f-4d6a-8f62-c35482526b63"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:04.927701 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:04.927675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d0542d-1a2f-4d6a-8f62-c35482526b63-kube-api-access-2lw6p" (OuterVolumeSpecName: "kube-api-access-2lw6p") pod "34d0542d-1a2f-4d6a-8f62-c35482526b63" (UID: "34d0542d-1a2f-4d6a-8f62-c35482526b63"). InnerVolumeSpecName "kube-api-access-2lw6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:04.930228 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:04.930195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-util" (OuterVolumeSpecName: "util") pod "34d0542d-1a2f-4d6a-8f62-c35482526b63" (UID: "34d0542d-1a2f-4d6a-8f62-c35482526b63"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:05.026216 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:05.026119    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-util\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:46:05.026216 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:05.026151    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lw6p\" (UniqueName: \"kubernetes.io/projected/34d0542d-1a2f-4d6a-8f62-c35482526b63-kube-api-access-2lw6p\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:46:05.026216 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:05.026162    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d0542d-1a2f-4d6a-8f62-c35482526b63-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:46:05.711448 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:05.711417    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g"
Sep 17 17:46:05.711638 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:05.711423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ar784g" event={"ID":"34d0542d-1a2f-4d6a-8f62-c35482526b63","Type":"ContainerDied","Data":"07585607bfb78652f1d3caecf0d7067dd40008329ea09e631d42a6aa0d1b119f"}
Sep 17 17:46:05.711638 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:05.711530    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07585607bfb78652f1d3caecf0d7067dd40008329ea09e631d42a6aa0d1b119f"
Sep 17 17:46:06.460896 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:06.460845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:07.908699 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:07.908667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:08.166261 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166169    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"]
Sep 17 17:46:08.166490 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166478    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerName="pull"
Sep 17 17:46:08.166531 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166492    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerName="pull"
Sep 17 17:46:08.166531 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166502    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerName="extract"
Sep 17 17:46:08.166531 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166508    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerName="extract"
Sep 17 17:46:08.166531 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166517    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerName="util"
Sep 17 17:46:08.166531 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166523    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerName="util"
Sep 17 17:46:08.166684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.166573    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d0542d-1a2f-4d6a-8f62-c35482526b63" containerName="extract"
Sep 17 17:46:08.170138 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.170116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.173187 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.173163    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:46:08.173361 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.173163    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-h86zz\""
Sep 17 17:46:08.173361 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.173325    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:46:08.173645 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.173625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:46:08.190532 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.190499    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"]
Sep 17 17:46:08.251375 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.251323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/721149d8-474b-47a1-947d-18f6e8fde230-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-q2mq8\" (UID: \"721149d8-474b-47a1-947d-18f6e8fde230\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.251375 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.251381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9twt\" (UniqueName: \"kubernetes.io/projected/721149d8-474b-47a1-947d-18f6e8fde230-kube-api-access-f9twt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-q2mq8\" (UID: \"721149d8-474b-47a1-947d-18f6e8fde230\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.352387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.352347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/721149d8-474b-47a1-947d-18f6e8fde230-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-q2mq8\" (UID: \"721149d8-474b-47a1-947d-18f6e8fde230\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.352550 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.352405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9twt\" (UniqueName: \"kubernetes.io/projected/721149d8-474b-47a1-947d-18f6e8fde230-kube-api-access-f9twt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-q2mq8\" (UID: \"721149d8-474b-47a1-947d-18f6e8fde230\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.354826 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.354801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/721149d8-474b-47a1-947d-18f6e8fde230-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-q2mq8\" (UID: \"721149d8-474b-47a1-947d-18f6e8fde230\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.362352 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.362320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9twt\" (UniqueName: \"kubernetes.io/projected/721149d8-474b-47a1-947d-18f6e8fde230-kube-api-access-f9twt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-q2mq8\" (UID: \"721149d8-474b-47a1-947d-18f6e8fde230\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.481601 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.481502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:08.770484 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:08.768568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"]
Sep 17 17:46:09.730895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:09.730835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8" event={"ID":"721149d8-474b-47a1-947d-18f6e8fde230","Type":"ContainerStarted","Data":"1feaeadb843b6ab38889fae1d283697b08e892545b81e7dd00b233c1ec85f3f9"}
Sep 17 17:46:11.733341 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:11.733298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:11.741608 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:11.741568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8" event={"ID":"721149d8-474b-47a1-947d-18f6e8fde230","Type":"ContainerStarted","Data":"3af88dfbfc7a4c161885f40a76c26ba13ee088ee2b88f4d620333268d016c84d"}
Sep 17 17:46:11.741821 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:11.741656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:11.768681 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:11.768624    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8" podStartSLOduration=1.273218406 podStartE2EDuration="3.768604957s" totalImagesPullingTime="2.495386551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:11.766115424 +0000 UTC m=+348.105509006" watchObservedRunningTime="2026-09-17 17:46:11.768604957 +0000 UTC m=+348.107998550"
Sep 17 17:46:12.032113 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.032002    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-b7ftl"]
Sep 17 17:46:12.034020 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.033988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:12.035608 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.035587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.039930 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.039904    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:46:12.040310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.040293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 17:46:12.040788 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.040766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-kbszq\""
Sep 17 17:46:12.047414 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.047385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-b7ftl"]
Sep 17 17:46:12.089664 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.089627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.089819 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.089680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/763aa643-7201-45e1-9c8f-2e01ed77f6ec-cabundle0\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.089819 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.089703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzx9x\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-kube-api-access-hzx9x\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.190332 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.190283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.190516 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.190345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/763aa643-7201-45e1-9c8f-2e01ed77f6ec-cabundle0\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.190516 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.190374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzx9x\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-kube-api-access-hzx9x\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.190516 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.190462    2580 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 17:46:12.190516 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.190492    2580 secret.go:281] references non-existent secret key: ca.crt
Sep 17 17:46:12.190516 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.190504    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 17:46:12.190516 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.190520    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-b7ftl: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 17:46:12.190756 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.190597    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates podName:763aa643-7201-45e1-9c8f-2e01ed77f6ec nodeName:}" failed. No retries permitted until 2026-09-17 17:46:12.690574823 +0000 UTC m=+349.029968405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates") pod "keda-operator-579c8958f7-b7ftl" (UID: "763aa643-7201-45e1-9c8f-2e01ed77f6ec") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 17:46:12.191219 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.191196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/763aa643-7201-45e1-9c8f-2e01ed77f6ec-cabundle0\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.208685 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.208656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzx9x\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-kube-api-access-hzx9x\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.240874 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.240831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:12.259115 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.259066    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"]
Sep 17 17:46:12.262594 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.262569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.265379 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.265350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 17:46:12.291180 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.291065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.291180 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.291166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6e56306e-7906-4ec4-8808-21bb494385c1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.291373 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.291243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tzp\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-kube-api-access-x8tzp\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.298271 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.298242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"]
Sep 17 17:46:12.391935 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.391890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.391935 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.391945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6e56306e-7906-4ec4-8808-21bb494385c1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.392229 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.391996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tzp\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-kube-api-access-x8tzp\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.392229 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.392042    2580 secret.go:281] references non-existent secret key: tls.crt
Sep 17 17:46:12.392229 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.392065    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 17:46:12.392229 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.392110    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2: references non-existent secret key: tls.crt
Sep 17 17:46:12.392229 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.392173    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates podName:6e56306e-7906-4ec4-8808-21bb494385c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:46:12.892156622 +0000 UTC m=+349.231550326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates") pod "keda-metrics-apiserver-77857b7b98-zwbj2" (UID: "6e56306e-7906-4ec4-8808-21bb494385c1") : references non-existent secret key: tls.crt
Sep 17 17:46:12.392434 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.392409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6e56306e-7906-4ec4-8808-21bb494385c1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.405650 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.405623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tzp\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-kube-api-access-x8tzp\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.420002 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.419965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:12.565589 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.565484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:12.694818 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.694771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:12.695020 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.694936    2580 secret.go:281] references non-existent secret key: ca.crt
Sep 17 17:46:12.695020 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.694965    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 17:46:12.695020 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.694975    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-b7ftl: references non-existent secret key: ca.crt
Sep 17 17:46:12.695177 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.695031    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates podName:763aa643-7201-45e1-9c8f-2e01ed77f6ec nodeName:}" failed. No retries permitted until 2026-09-17 17:46:13.695016717 +0000 UTC m=+350.034410286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates") pod "keda-operator-579c8958f7-b7ftl" (UID: "763aa643-7201-45e1-9c8f-2e01ed77f6ec") : references non-existent secret key: ca.crt
Sep 17 17:46:12.897274 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.897235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:12.897650 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.897363    2580 secret.go:281] references non-existent secret key: tls.crt
Sep 17 17:46:12.897650 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.897377    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 17:46:12.897650 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.897393    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2: references non-existent secret key: tls.crt
Sep 17 17:46:12.897650 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:46:12.897444    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates podName:6e56306e-7906-4ec4-8808-21bb494385c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:46:13.897431357 +0000 UTC m=+350.236824927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates") pod "keda-metrics-apiserver-77857b7b98-zwbj2" (UID: "6e56306e-7906-4ec4-8808-21bb494385c1") : references non-existent secret key: tls.crt
Sep 17 17:46:12.999490 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:12.999453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.108142 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.108108    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-xg8dw"]
Sep 17 17:46:13.111482 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.111463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.114093 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.114061    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 17:46:13.128116 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.128063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-xg8dw"]
Sep 17 17:46:13.160597 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.160502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.200518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.200480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46b46481-e240-4c50-886e-462800aa0ae0-certificates\") pod \"keda-admission-6c4d864b86-xg8dw\" (UID: \"46b46481-e240-4c50-886e-462800aa0ae0\") " pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.200711 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.200552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drr87\" (UniqueName: \"kubernetes.io/projected/46b46481-e240-4c50-886e-462800aa0ae0-kube-api-access-drr87\") pod \"keda-admission-6c4d864b86-xg8dw\" (UID: \"46b46481-e240-4c50-886e-462800aa0ae0\") " pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.263069 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.263021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.301066 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.301022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drr87\" (UniqueName: \"kubernetes.io/projected/46b46481-e240-4c50-886e-462800aa0ae0-kube-api-access-drr87\") pod \"keda-admission-6c4d864b86-xg8dw\" (UID: \"46b46481-e240-4c50-886e-462800aa0ae0\") " pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.301253 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.301130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46b46481-e240-4c50-886e-462800aa0ae0-certificates\") pod \"keda-admission-6c4d864b86-xg8dw\" (UID: \"46b46481-e240-4c50-886e-462800aa0ae0\") " pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.303743 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.303724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46b46481-e240-4c50-886e-462800aa0ae0-certificates\") pod \"keda-admission-6c4d864b86-xg8dw\" (UID: \"46b46481-e240-4c50-886e-462800aa0ae0\") " pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.313401 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.313369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drr87\" (UniqueName: \"kubernetes.io/projected/46b46481-e240-4c50-886e-462800aa0ae0-kube-api-access-drr87\") pod \"keda-admission-6c4d864b86-xg8dw\" (UID: \"46b46481-e240-4c50-886e-462800aa0ae0\") " pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.428134 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.428009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:13.704875 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.704773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:13.708367 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.708293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/763aa643-7201-45e1-9c8f-2e01ed77f6ec-certificates\") pod \"keda-operator-579c8958f7-b7ftl\" (UID: \"763aa643-7201-45e1-9c8f-2e01ed77f6ec\") " pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:13.708976 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.708949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-xg8dw"]
Sep 17 17:46:13.711662 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.711629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.712608 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:46:13.712580    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b46481_e240_4c50_886e_462800aa0ae0.slice/crio-77370c16789ec597383f3f529ee45528c71cc5109202297950b739f51ab91804 WatchSource:0}: Error finding container 77370c16789ec597383f3f529ee45528c71cc5109202297950b739f51ab91804: Status 404 returned error can't find the container with id 77370c16789ec597383f3f529ee45528c71cc5109202297950b739f51ab91804
Sep 17 17:46:13.752069 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.752033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-xg8dw" event={"ID":"46b46481-e240-4c50-886e-462800aa0ae0","Type":"ContainerStarted","Data":"77370c16789ec597383f3f529ee45528c71cc5109202297950b739f51ab91804"}
Sep 17 17:46:13.847435 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.847395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:13.879149 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.879111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.907169 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.907125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:13.912587 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:13.912553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6e56306e-7906-4ec4-8808-21bb494385c1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-zwbj2\" (UID: \"6e56306e-7906-4ec4-8808-21bb494385c1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:14.040066 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:14.039957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:14.073403 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:14.073363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:14.132386 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:14.132342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-b7ftl"]
Sep 17 17:46:14.134710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:14.134605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:14.137185 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:46:14.136364    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763aa643_7201_45e1_9c8f_2e01ed77f6ec.slice/crio-a4103acd1f4bfc773ae391c3366442cd634cbc2b20148a5187c5649e9205766d WatchSource:0}: Error finding container a4103acd1f4bfc773ae391c3366442cd634cbc2b20148a5187c5649e9205766d: Status 404 returned error can't find the container with id a4103acd1f4bfc773ae391c3366442cd634cbc2b20148a5187c5649e9205766d
Sep 17 17:46:14.357424 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:14.357394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"]
Sep 17 17:46:14.359909 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:46:14.359872    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e56306e_7906_4ec4_8808_21bb494385c1.slice/crio-9bd2f29e83748b21c27f566db826e503913585c5bd82560d3c8a60205c697c2d WatchSource:0}: Error finding container 9bd2f29e83748b21c27f566db826e503913585c5bd82560d3c8a60205c697c2d: Status 404 returned error can't find the container with id 9bd2f29e83748b21c27f566db826e503913585c5bd82560d3c8a60205c697c2d
Sep 17 17:46:14.758635 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:14.758597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-b7ftl" event={"ID":"763aa643-7201-45e1-9c8f-2e01ed77f6ec","Type":"ContainerStarted","Data":"a4103acd1f4bfc773ae391c3366442cd634cbc2b20148a5187c5649e9205766d"}
Sep 17 17:46:14.760078 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:14.760049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2" event={"ID":"6e56306e-7906-4ec4-8808-21bb494385c1","Type":"ContainerStarted","Data":"9bd2f29e83748b21c27f566db826e503913585c5bd82560d3c8a60205c697c2d"}
Sep 17 17:46:15.768841 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:15.768788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-xg8dw" event={"ID":"46b46481-e240-4c50-886e-462800aa0ae0","Type":"ContainerStarted","Data":"4546af4f48375d64bbc313280e194eed8e3e5cead1f2df25c3b845e66e3e3954"}
Sep 17 17:46:15.769360 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:15.769055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:15.795931 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:15.795858    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-xg8dw" podStartSLOduration=1.242809297 podStartE2EDuration="2.795835007s" totalImagesPullingTime="1.55302571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:15.794018212 +0000 UTC m=+352.133411827" watchObservedRunningTime="2026-09-17 17:46:15.795835007 +0000 UTC m=+352.135228599"
Sep 17 17:46:18.786794 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:18.786697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2" event={"ID":"6e56306e-7906-4ec4-8808-21bb494385c1","Type":"ContainerStarted","Data":"701717a5338acbc0816fd106a11819f05dd945bb9cd6c241bf04e49494be75d1"}
Sep 17 17:46:18.787278 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:18.786813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:18.788623 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:18.788590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-b7ftl" event={"ID":"763aa643-7201-45e1-9c8f-2e01ed77f6ec","Type":"ContainerStarted","Data":"942f8bcbf0326c619e4fc6240d6f9dd8e31fc146c31dd01b588401e77ba11f2b"}
Sep 17 17:46:18.788752 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:18.788675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:46:18.806514 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:18.806454    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2" podStartSLOduration=2.765269425 podStartE2EDuration="6.806434481s" totalImagesPullingTime="4.041165056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:18.803893558 +0000 UTC m=+355.143287148" watchObservedRunningTime="2026-09-17 17:46:18.806434481 +0000 UTC m=+355.145828073"
Sep 17 17:46:18.823827 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:18.823772    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-b7ftl" podStartSLOduration=2.554257296 podStartE2EDuration="6.823753606s" totalImagesPullingTime="4.26949631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:18.821870856 +0000 UTC m=+355.161264448" watchObservedRunningTime="2026-09-17 17:46:18.823753606 +0000 UTC m=+355.163147197"
Sep 17 17:46:28.834003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:28.833940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:28.903330 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:28.903287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:29.797589 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:29.797552    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-zwbj2"
Sep 17 17:46:32.748985 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:32.748955    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-q2mq8"
Sep 17 17:46:36.776923 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:36.776882    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-xg8dw"
Sep 17 17:46:39.795526 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:46:39.795495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-b7ftl"
Sep 17 17:47:01.375946 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:01.375904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:01.476288 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:01.476253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:04.439805 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.439768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:04.540697 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.540653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:04.716385 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.716303    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"]
Sep 17 17:47:04.725359 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.725334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.728214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.728160    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:47:04.729339 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.729300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ndbbk\""
Sep 17 17:47:04.729469 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.729300    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:47:04.731238 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.731209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"]
Sep 17 17:47:04.854889 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.854853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdd5d\" (UniqueName: \"kubernetes.io/projected/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-kube-api-access-cdd5d\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.855138 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.854923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.855138 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.854983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.955969 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.955927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.955969 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.955974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.956225 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.956014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdd5d\" (UniqueName: \"kubernetes.io/projected/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-kube-api-access-cdd5d\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.956462 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.956435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.956500 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.956445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:04.966560 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:04.966489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdd5d\" (UniqueName: \"kubernetes.io/projected/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-kube-api-access-cdd5d\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:05.036254 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:05.036217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:05.314844 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:05.314742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"]
Sep 17 17:47:05.319140 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:47:05.319071    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfb5f81_ba14_439c_8415_860cc4cf9ce5.slice/crio-798d0094f811c8a11bb3dccca0e9404f2bf47ab4fcfa7975da5d707d6c417b38 WatchSource:0}: Error finding container 798d0094f811c8a11bb3dccca0e9404f2bf47ab4fcfa7975da5d707d6c417b38: Status 404 returned error can't find the container with id 798d0094f811c8a11bb3dccca0e9404f2bf47ab4fcfa7975da5d707d6c417b38
Sep 17 17:47:06.017044 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:06.017010    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerID="84bdbc0341bee4cf31febfaf5fbc506270c53870af227df4b85981b71b52a00a" exitCode=0
Sep 17 17:47:06.017613 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:06.017059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c" event={"ID":"dcfb5f81-ba14-439c-8415-860cc4cf9ce5","Type":"ContainerDied","Data":"84bdbc0341bee4cf31febfaf5fbc506270c53870af227df4b85981b71b52a00a"}
Sep 17 17:47:06.017613 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:06.017112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c" event={"ID":"dcfb5f81-ba14-439c-8415-860cc4cf9ce5","Type":"ContainerStarted","Data":"798d0094f811c8a11bb3dccca0e9404f2bf47ab4fcfa7975da5d707d6c417b38"}
Sep 17 17:47:09.033575 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:09.033531    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerID="cef8c4b48f074036c087a12025aa9d8840d94a03374bf5ff837546b54ddcf3f0" exitCode=0
Sep 17 17:47:09.033977 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:09.033619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c" event={"ID":"dcfb5f81-ba14-439c-8415-860cc4cf9ce5","Type":"ContainerDied","Data":"cef8c4b48f074036c087a12025aa9d8840d94a03374bf5ff837546b54ddcf3f0"}
Sep 17 17:47:10.040769 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:10.040729    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerID="40eb3fd104c8e3a38160d445b929f7324f78c1e887432e7ef7023290936c49bc" exitCode=0
Sep 17 17:47:10.044923 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:10.044887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c" event={"ID":"dcfb5f81-ba14-439c-8415-860cc4cf9ce5","Type":"ContainerDied","Data":"40eb3fd104c8e3a38160d445b929f7324f78c1e887432e7ef7023290936c49bc"}
Sep 17 17:47:11.179741 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.179709    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:11.316280 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.316196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-util\") pod \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") "
Sep 17 17:47:11.316280 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.316262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdd5d\" (UniqueName: \"kubernetes.io/projected/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-kube-api-access-cdd5d\") pod \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") "
Sep 17 17:47:11.316510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.316361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-bundle\") pod \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\" (UID: \"dcfb5f81-ba14-439c-8415-860cc4cf9ce5\") "
Sep 17 17:47:11.317337 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.317300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-bundle" (OuterVolumeSpecName: "bundle") pod "dcfb5f81-ba14-439c-8415-860cc4cf9ce5" (UID: "dcfb5f81-ba14-439c-8415-860cc4cf9ce5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:11.318577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.318543    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-kube-api-access-cdd5d" (OuterVolumeSpecName: "kube-api-access-cdd5d") pod "dcfb5f81-ba14-439c-8415-860cc4cf9ce5" (UID: "dcfb5f81-ba14-439c-8415-860cc4cf9ce5"). InnerVolumeSpecName "kube-api-access-cdd5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:11.322172 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.322060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-util" (OuterVolumeSpecName: "util") pod "dcfb5f81-ba14-439c-8415-860cc4cf9ce5" (UID: "dcfb5f81-ba14-439c-8415-860cc4cf9ce5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:11.417645 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.417602    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:47:11.417645 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.417636    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-util\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:47:11.417645 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:11.417646    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdd5d\" (UniqueName: \"kubernetes.io/projected/dcfb5f81-ba14-439c-8415-860cc4cf9ce5-kube-api-access-cdd5d\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:47:12.051289 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:12.051250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c" event={"ID":"dcfb5f81-ba14-439c-8415-860cc4cf9ce5","Type":"ContainerDied","Data":"798d0094f811c8a11bb3dccca0e9404f2bf47ab4fcfa7975da5d707d6c417b38"}
Sep 17 17:47:12.051483 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:12.051303    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798d0094f811c8a11bb3dccca0e9404f2bf47ab4fcfa7975da5d707d6c417b38"
Sep 17 17:47:12.051483 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:12.051260    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mx8c"
Sep 17 17:47:15.897860 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:15.897824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:16.137583 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:16.137548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:18.082059 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082021    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"]
Sep 17 17:47:18.082488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082385    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerName="util"
Sep 17 17:47:18.082488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082397    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerName="util"
Sep 17 17:47:18.082488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082411    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerName="pull"
Sep 17 17:47:18.082488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082417    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerName="pull"
Sep 17 17:47:18.082488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082429    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerName="extract"
Sep 17 17:47:18.082488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082436    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerName="extract"
Sep 17 17:47:18.082488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.082488    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcfb5f81-ba14-439c-8415-860cc4cf9ce5" containerName="extract"
Sep 17 17:47:18.085504 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.085487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.088927 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.088901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 17:47:18.089057 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.088983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6skk9\""
Sep 17 17:47:18.089114 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.089060    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:47:18.104826 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.104794    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"]
Sep 17 17:47:18.180892 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.180850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13440ad8-4a1b-46a3-8c42-661771edc37d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tvxg2\" (UID: \"13440ad8-4a1b-46a3-8c42-661771edc37d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.181117 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.180899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkn7\" (UniqueName: \"kubernetes.io/projected/13440ad8-4a1b-46a3-8c42-661771edc37d-kube-api-access-qxkn7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tvxg2\" (UID: \"13440ad8-4a1b-46a3-8c42-661771edc37d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.282305 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.282264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13440ad8-4a1b-46a3-8c42-661771edc37d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tvxg2\" (UID: \"13440ad8-4a1b-46a3-8c42-661771edc37d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.282305 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.282320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkn7\" (UniqueName: \"kubernetes.io/projected/13440ad8-4a1b-46a3-8c42-661771edc37d-kube-api-access-qxkn7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tvxg2\" (UID: \"13440ad8-4a1b-46a3-8c42-661771edc37d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.282710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.282682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13440ad8-4a1b-46a3-8c42-661771edc37d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tvxg2\" (UID: \"13440ad8-4a1b-46a3-8c42-661771edc37d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.291936 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.291908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkn7\" (UniqueName: \"kubernetes.io/projected/13440ad8-4a1b-46a3-8c42-661771edc37d-kube-api-access-qxkn7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tvxg2\" (UID: \"13440ad8-4a1b-46a3-8c42-661771edc37d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.395384 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.395353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"
Sep 17 17:47:18.685923 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:18.685896    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2"]
Sep 17 17:47:18.688896 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:47:18.688861    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13440ad8_4a1b_46a3_8c42_661771edc37d.slice/crio-5d7e51f356cd8ad9d7d8b42e5316af0cab77c218db93d95e7a76f7aa9cbf595f WatchSource:0}: Error finding container 5d7e51f356cd8ad9d7d8b42e5316af0cab77c218db93d95e7a76f7aa9cbf595f: Status 404 returned error can't find the container with id 5d7e51f356cd8ad9d7d8b42e5316af0cab77c218db93d95e7a76f7aa9cbf595f
Sep 17 17:47:19.083908 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:19.083797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2" event={"ID":"13440ad8-4a1b-46a3-8c42-661771edc37d","Type":"ContainerStarted","Data":"5d7e51f356cd8ad9d7d8b42e5316af0cab77c218db93d95e7a76f7aa9cbf595f"}
Sep 17 17:47:19.931213 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:19.931167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:20.029384 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:20.029336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:21.095932 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:21.095891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2" event={"ID":"13440ad8-4a1b-46a3-8c42-661771edc37d","Type":"ContainerStarted","Data":"3e486374f26c7a83fc9aef5dd6dde0c9a4305ae7046fa088373f9b2ab497d11a"}
Sep 17 17:47:21.134206 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:21.134134    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tvxg2" podStartSLOduration=1.400028402 podStartE2EDuration="3.134109907s" totalImagesPullingTime="1.734081505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:21.133693631 +0000 UTC m=+417.473087223" watchObservedRunningTime="2026-09-17 17:47:21.134109907 +0000 UTC m=+417.473503500"
Sep 17 17:47:21.390592 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:21.390552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:21.492551 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:21.492507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:23.766043 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:23.766010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:24.179645 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:24.179609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:24.730149 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:24.730110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:24.836242 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:24.836201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:25.852945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:25.852903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:25.956476 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:25.956429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:26.056621 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.056583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:26.820051 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.820014    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"]
Sep 17 17:47:26.823736 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.823716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:26.826418 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.826394    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:47:26.826693 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.826675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ndbbk\""
Sep 17 17:47:26.827677 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.827656    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:47:26.836762 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.836735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"]
Sep 17 17:47:26.956580 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.956534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:26.956995 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.956596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gll\" (UniqueName: \"kubernetes.io/projected/504033dc-2611-4bf9-b542-f228d05a10c7-kube-api-access-c8gll\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:26.956995 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:26.956730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.057270 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.057228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gll\" (UniqueName: \"kubernetes.io/projected/504033dc-2611-4bf9-b542-f228d05a10c7-kube-api-access-c8gll\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.057437 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.057310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.057437 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.057335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.057706 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.057690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.057751 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.057720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.069008 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.068974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gll\" (UniqueName: \"kubernetes.io/projected/504033dc-2611-4bf9-b542-f228d05a10c7-kube-api-access-c8gll\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.132852 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.132766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:27.409110 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:27.409066    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"]
Sep 17 17:47:27.411617 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:47:27.411580    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504033dc_2611_4bf9_b542_f228d05a10c7.slice/crio-b785ef21acdac39019ba7f934d57210e169ab9b520ba2c9e815d558131362dfe WatchSource:0}: Error finding container b785ef21acdac39019ba7f934d57210e169ab9b520ba2c9e815d558131362dfe: Status 404 returned error can't find the container with id b785ef21acdac39019ba7f934d57210e169ab9b520ba2c9e815d558131362dfe
Sep 17 17:47:28.131130 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:28.131075    2580 generic.go:356] "Generic (PLEG): container finished" podID="504033dc-2611-4bf9-b542-f228d05a10c7" containerID="85385f332e18450c439af579c268bec8edfb4370fb0a715c73302b8ca780a054" exitCode=0
Sep 17 17:47:28.131480 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:28.131158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf" event={"ID":"504033dc-2611-4bf9-b542-f228d05a10c7","Type":"ContainerDied","Data":"85385f332e18450c439af579c268bec8edfb4370fb0a715c73302b8ca780a054"}
Sep 17 17:47:28.131480 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:28.131206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf" event={"ID":"504033dc-2611-4bf9-b542-f228d05a10c7","Type":"ContainerStarted","Data":"b785ef21acdac39019ba7f934d57210e169ab9b520ba2c9e815d558131362dfe"}
Sep 17 17:47:31.147404 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:31.147367    2580 generic.go:356] "Generic (PLEG): container finished" podID="504033dc-2611-4bf9-b542-f228d05a10c7" containerID="5705fa3a3ffc3e55c401a7ea2c58c0e37e1ab7259173c4c45defc9954402e54c" exitCode=0
Sep 17 17:47:31.147802 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:31.147460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf" event={"ID":"504033dc-2611-4bf9-b542-f228d05a10c7","Type":"ContainerDied","Data":"5705fa3a3ffc3e55c401a7ea2c58c0e37e1ab7259173c4c45defc9954402e54c"}
Sep 17 17:47:31.972320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:31.972272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:32.154468 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:32.154434    2580 generic.go:356] "Generic (PLEG): container finished" podID="504033dc-2611-4bf9-b542-f228d05a10c7" containerID="5d8abdbf00ed4e324bd7d8377c0b79cd99546637a1b1d936a4c8b90d34fd2975" exitCode=0
Sep 17 17:47:32.154875 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:32.154522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf" event={"ID":"504033dc-2611-4bf9-b542-f228d05a10c7","Type":"ContainerDied","Data":"5d8abdbf00ed4e324bd7d8377c0b79cd99546637a1b1d936a4c8b90d34fd2975"}
Sep 17 17:47:33.294169 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.294143    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:33.415189 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.415065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-util\") pod \"504033dc-2611-4bf9-b542-f228d05a10c7\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") "
Sep 17 17:47:33.415189 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.415165    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-bundle\") pod \"504033dc-2611-4bf9-b542-f228d05a10c7\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") "
Sep 17 17:47:33.415189 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.415190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8gll\" (UniqueName: \"kubernetes.io/projected/504033dc-2611-4bf9-b542-f228d05a10c7-kube-api-access-c8gll\") pod \"504033dc-2611-4bf9-b542-f228d05a10c7\" (UID: \"504033dc-2611-4bf9-b542-f228d05a10c7\") "
Sep 17 17:47:33.415559 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.415527    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-bundle" (OuterVolumeSpecName: "bundle") pod "504033dc-2611-4bf9-b542-f228d05a10c7" (UID: "504033dc-2611-4bf9-b542-f228d05a10c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:33.417395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.417372    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504033dc-2611-4bf9-b542-f228d05a10c7-kube-api-access-c8gll" (OuterVolumeSpecName: "kube-api-access-c8gll") pod "504033dc-2611-4bf9-b542-f228d05a10c7" (UID: "504033dc-2611-4bf9-b542-f228d05a10c7"). InnerVolumeSpecName "kube-api-access-c8gll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:33.421712 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.421681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-util" (OuterVolumeSpecName: "util") pod "504033dc-2611-4bf9-b542-f228d05a10c7" (UID: "504033dc-2611-4bf9-b542-f228d05a10c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:33.516431 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.516387    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:47:33.516431 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.516419    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gll\" (UniqueName: \"kubernetes.io/projected/504033dc-2611-4bf9-b542-f228d05a10c7-kube-api-access-c8gll\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:47:33.516431 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.516431    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504033dc-2611-4bf9-b542-f228d05a10c7-util\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:47:33.887515 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.887419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:33.988146 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:33.988079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:34.165820 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:34.165784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf" event={"ID":"504033dc-2611-4bf9-b542-f228d05a10c7","Type":"ContainerDied","Data":"b785ef21acdac39019ba7f934d57210e169ab9b520ba2c9e815d558131362dfe"}
Sep 17 17:47:34.165820 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:34.165822    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8clf"
Sep 17 17:47:34.166031 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:34.165837    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b785ef21acdac39019ba7f934d57210e169ab9b520ba2c9e815d558131362dfe"
Sep 17 17:47:39.451340 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:39.451302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:39.538660 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:39.538624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:44.720219 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:44.720143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:44.856138 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:44.856104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:44.954938 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:44.954906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:45.268878 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:45.268846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:45.372550 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:45.372513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:49.580218 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:49.580185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:54.297150 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.297114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:54.316235 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316200    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"]
Sep 17 17:47:54.316581 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316568    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504033dc-2611-4bf9-b542-f228d05a10c7" containerName="extract"
Sep 17 17:47:54.316629 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316583    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="504033dc-2611-4bf9-b542-f228d05a10c7" containerName="extract"
Sep 17 17:47:54.316629 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316591    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504033dc-2611-4bf9-b542-f228d05a10c7" containerName="util"
Sep 17 17:47:54.316629 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316596    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="504033dc-2611-4bf9-b542-f228d05a10c7" containerName="util"
Sep 17 17:47:54.316629 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316611    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504033dc-2611-4bf9-b542-f228d05a10c7" containerName="pull"
Sep 17 17:47:54.316629 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316619    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="504033dc-2611-4bf9-b542-f228d05a10c7" containerName="pull"
Sep 17 17:47:54.316783 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.316707    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="504033dc-2611-4bf9-b542-f228d05a10c7" containerName="extract"
Sep 17 17:47:54.319912 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.319893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.327805 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.322947    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 17:47:54.327805 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.323399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s5vcf\""
Sep 17 17:47:54.327805 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.323630    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 17:47:54.327805 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.323861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:47:54.327805 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.324056    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 17:47:54.334272 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.334240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"]
Sep 17 17:47:54.367324 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.367283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:54.381932 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.381891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.382149 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.381949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.382149 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.381986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.382149 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.382009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.382149 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.382053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f98e88-f419-4159-a365-83f338cf10f9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.382149 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.382135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75wr\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-kube-api-access-s75wr\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.382364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.382180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.483626 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.483590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.483626 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.483628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.483872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.483645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.483872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.483671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f98e88-f419-4159-a365-83f338cf10f9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.483872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.483698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75wr\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-kube-api-access-s75wr\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.483872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.483738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.483872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.483787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.484606 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.484577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.486343 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.486319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f98e88-f419-4159-a365-83f338cf10f9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.486343 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.486332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.486479 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.486357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.486479 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.486446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.493365 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.493333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.493610 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.493585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75wr\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-kube-api-access-s75wr\") pod \"istiod-openshift-gateway-7b7995d5cd-l7z8h\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.633461 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.633351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:47:54.915558 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:54.915511    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"]
Sep 17 17:47:54.920105 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:47:54.920020    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f98e88_f419_4159_a365_83f338cf10f9.slice/crio-368aac0fdae6173673453604f48b98867a39ab2d5dea6813513bae2be205cb8f WatchSource:0}: Error finding container 368aac0fdae6173673453604f48b98867a39ab2d5dea6813513bae2be205cb8f: Status 404 returned error can't find the container with id 368aac0fdae6173673453604f48b98867a39ab2d5dea6813513bae2be205cb8f
Sep 17 17:47:55.266589 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:55.266498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h" event={"ID":"97f98e88-f419-4159-a365-83f338cf10f9","Type":"ContainerStarted","Data":"368aac0fdae6173673453604f48b98867a39ab2d5dea6813513bae2be205cb8f"}
Sep 17 17:47:55.596110 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:47:55.595977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:00.159869 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:00.159827    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:48:00.160180 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:00.159902    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:48:00.294057 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:00.294018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h" event={"ID":"97f98e88-f419-4159-a365-83f338cf10f9","Type":"ContainerStarted","Data":"2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5"}
Sep 17 17:48:00.294291 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:00.294121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:48:00.320527 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:00.320459    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h" podStartSLOduration=1.085352564 podStartE2EDuration="6.320438677s" totalImagesPullingTime="5.235086113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:00.318111528 +0000 UTC m=+456.657505119" watchObservedRunningTime="2026-09-17 17:48:00.320438677 +0000 UTC m=+456.659832268"
Sep 17 17:48:00.643136 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:00.643077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:00.788448 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:00.788412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:01.302224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:01.302189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:48:01.890382 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:01.890327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:02.058520 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:02.058421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:05.017785 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:05.017746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:07.325668 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:07.325632    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:07.500507 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:07.500465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:12.048959 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.048918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:12.163437 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.163390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:12.332439 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.332351    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"]
Sep 17 17:48:12.336094 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.336065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.339162 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.339137    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ndbbk\""
Sep 17 17:48:12.339307 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.339165    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:48:12.339968 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.339952    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:48:12.358996 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.358958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"]
Sep 17 17:48:12.431423 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.431379    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"]
Sep 17 17:48:12.435026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.435007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.439262 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.439233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb26v\" (UniqueName: \"kubernetes.io/projected/14b1708e-46f9-4ff7-9429-866a62e9d8c9-kube-api-access-bb26v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.439416 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.439272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.439416 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.439300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.446103 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.446058    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"]
Sep 17 17:48:12.493398 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.493360    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"]
Sep 17 17:48:12.497219 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.497194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.507561 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.507530    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"]
Sep 17 17:48:12.539788 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.539788 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.540011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8vc\" (UniqueName: \"kubernetes.io/projected/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-kube-api-access-tg8vc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.540011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghmc\" (UniqueName: \"kubernetes.io/projected/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-kube-api-access-7ghmc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.540011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.540011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.540011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.540011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.539992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.540228 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.540017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb26v\" (UniqueName: \"kubernetes.io/projected/14b1708e-46f9-4ff7-9429-866a62e9d8c9-kube-api-access-bb26v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.540228 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.540207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.540293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.540229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.549146 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.549112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb26v\" (UniqueName: \"kubernetes.io/projected/14b1708e-46f9-4ff7-9429-866a62e9d8c9-kube-api-access-bb26v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.593163 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.593041    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"]
Sep 17 17:48:12.596792 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.596768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.607069 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.607042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"]
Sep 17 17:48:12.640735 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.640735 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.641003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.641003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.641003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62s5\" (UniqueName: \"kubernetes.io/projected/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-kube-api-access-x62s5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.641003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.641003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8vc\" (UniqueName: \"kubernetes.io/projected/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-kube-api-access-tg8vc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.641003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.640975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.641310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.641021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghmc\" (UniqueName: \"kubernetes.io/projected/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-kube-api-access-7ghmc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.641310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.641201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.641310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.641232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.641310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.641245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.641479 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.641330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.646540 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.646521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:12.650560 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.650541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghmc\" (UniqueName: \"kubernetes.io/projected/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-kube-api-access-7ghmc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.650619 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.650557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8vc\" (UniqueName: \"kubernetes.io/projected/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-kube-api-access-tg8vc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.742528 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.742489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62s5\" (UniqueName: \"kubernetes.io/projected/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-kube-api-access-x62s5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.742528 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.742532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.742806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.742551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.742982 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.742962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.743054 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.742965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.745339 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.745316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:12.752211 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.752183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62s5\" (UniqueName: \"kubernetes.io/projected/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-kube-api-access-x62s5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.807019 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.806981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:12.907248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.907213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:12.923478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:12.923444    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"]
Sep 17 17:48:12.926652 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:48:12.926624    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b1708e_46f9_4ff7_9429_866a62e9d8c9.slice/crio-2a0a1b1675a416eedce2ad43f5693b0cdf46ea597d2f898f71a9f3fe8e8561af WatchSource:0}: Error finding container 2a0a1b1675a416eedce2ad43f5693b0cdf46ea597d2f898f71a9f3fe8e8561af: Status 404 returned error can't find the container with id 2a0a1b1675a416eedce2ad43f5693b0cdf46ea597d2f898f71a9f3fe8e8561af
Sep 17 17:48:13.025856 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.025827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"]
Sep 17 17:48:13.028673 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:48:13.028646    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90f5ee0_7960_4fa5_9adc_14175ca1a8cc.slice/crio-0680b237a96a90aba489fb3fe02a65873646081af01068c3b5d3e2b1813567c9 WatchSource:0}: Error finding container 0680b237a96a90aba489fb3fe02a65873646081af01068c3b5d3e2b1813567c9: Status 404 returned error can't find the container with id 0680b237a96a90aba489fb3fe02a65873646081af01068c3b5d3e2b1813567c9
Sep 17 17:48:13.088560 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.088529    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"]
Sep 17 17:48:13.103849 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:48:13.103811    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7ff20f_9a22_4eaa_a8fd_c2766eceb88a.slice/crio-09ba5bc8af9fe8cd6b2bf66ca7ddf29007f7d4e33bcf9b2741d4db99d26d30ed WatchSource:0}: Error finding container 09ba5bc8af9fe8cd6b2bf66ca7ddf29007f7d4e33bcf9b2741d4db99d26d30ed: Status 404 returned error can't find the container with id 09ba5bc8af9fe8cd6b2bf66ca7ddf29007f7d4e33bcf9b2741d4db99d26d30ed
Sep 17 17:48:13.191239 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.191206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"]
Sep 17 17:48:13.219207 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:48:13.219163    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc324ba_cf3e_4d87_8180_6e3b6c813ffe.slice/crio-acb303576653b937d34d8a810f8f5850386b7e743b6dea468a7b53a168cf8758 WatchSource:0}: Error finding container acb303576653b937d34d8a810f8f5850386b7e743b6dea468a7b53a168cf8758: Status 404 returned error can't find the container with id acb303576653b937d34d8a810f8f5850386b7e743b6dea468a7b53a168cf8758
Sep 17 17:48:13.364708 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.364616    2580 generic.go:356] "Generic (PLEG): container finished" podID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerID="faaffd8d4b157798c9d572334a92f4b371282ee8559b905b809f858604bd3cb1" exitCode=0
Sep 17 17:48:13.364875 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.364697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb" event={"ID":"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc","Type":"ContainerDied","Data":"faaffd8d4b157798c9d572334a92f4b371282ee8559b905b809f858604bd3cb1"}
Sep 17 17:48:13.364875 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.364794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb" event={"ID":"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc","Type":"ContainerStarted","Data":"0680b237a96a90aba489fb3fe02a65873646081af01068c3b5d3e2b1813567c9"}
Sep 17 17:48:13.366402 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.366380    2580 generic.go:356] "Generic (PLEG): container finished" podID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerID="786875ae638beee26837b201b6976eb34faa656a3f7bce29aa24935102b3d98a" exitCode=0
Sep 17 17:48:13.366484 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.366462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc" event={"ID":"edc324ba-cf3e-4d87-8180-6e3b6c813ffe","Type":"ContainerDied","Data":"786875ae638beee26837b201b6976eb34faa656a3f7bce29aa24935102b3d98a"}
Sep 17 17:48:13.366536 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.366504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc" event={"ID":"edc324ba-cf3e-4d87-8180-6e3b6c813ffe","Type":"ContainerStarted","Data":"acb303576653b937d34d8a810f8f5850386b7e743b6dea468a7b53a168cf8758"}
Sep 17 17:48:13.368487 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.368464    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerID="79497e5e97b4043310f2055304647d286ec3084aa69bbc6643c4807cbe3dc171" exitCode=0
Sep 17 17:48:13.368570 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.368521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g" event={"ID":"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a","Type":"ContainerDied","Data":"79497e5e97b4043310f2055304647d286ec3084aa69bbc6643c4807cbe3dc171"}
Sep 17 17:48:13.368620 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.368578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g" event={"ID":"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a","Type":"ContainerStarted","Data":"09ba5bc8af9fe8cd6b2bf66ca7ddf29007f7d4e33bcf9b2741d4db99d26d30ed"}
Sep 17 17:48:13.370496 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.370476    2580 generic.go:356] "Generic (PLEG): container finished" podID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerID="5389989bce0e303f4a5cd835f16a3bcc2f7392fc2bd0c6d39432e94980dfd3e8" exitCode=0
Sep 17 17:48:13.370593 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.370538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth" event={"ID":"14b1708e-46f9-4ff7-9429-866a62e9d8c9","Type":"ContainerDied","Data":"5389989bce0e303f4a5cd835f16a3bcc2f7392fc2bd0c6d39432e94980dfd3e8"}
Sep 17 17:48:13.370593 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:13.370569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth" event={"ID":"14b1708e-46f9-4ff7-9429-866a62e9d8c9","Type":"ContainerStarted","Data":"2a0a1b1675a416eedce2ad43f5693b0cdf46ea597d2f898f71a9f3fe8e8561af"}
Sep 17 17:48:14.377816 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:14.377779    2580 generic.go:356] "Generic (PLEG): container finished" podID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerID="f1805195bad4e855cc5e3138c8990a8d3a59198ebfeb4cae9d7e23d2cf123975" exitCode=0
Sep 17 17:48:14.378229 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:14.377858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb" event={"ID":"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc","Type":"ContainerDied","Data":"f1805195bad4e855cc5e3138c8990a8d3a59198ebfeb4cae9d7e23d2cf123975"}
Sep 17 17:48:15.385522 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:15.385487    2580 generic.go:356] "Generic (PLEG): container finished" podID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerID="7834be9d2bb4476de9fb2213f092c4872deba3f37d3395f06c38e597de2187cf" exitCode=0
Sep 17 17:48:15.385926 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:15.385573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb" event={"ID":"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc","Type":"ContainerDied","Data":"7834be9d2bb4476de9fb2213f092c4872deba3f37d3395f06c38e597de2187cf"}
Sep 17 17:48:15.387454 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:15.387429    2580 generic.go:356] "Generic (PLEG): container finished" podID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerID="54b6c9804f4381e315111d0b01e0289072f2788dd25822f1aa48eb5914ef9061" exitCode=0
Sep 17 17:48:15.387592 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:15.387515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc" event={"ID":"edc324ba-cf3e-4d87-8180-6e3b6c813ffe","Type":"ContainerDied","Data":"54b6c9804f4381e315111d0b01e0289072f2788dd25822f1aa48eb5914ef9061"}
Sep 17 17:48:15.389577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:15.389558    2580 generic.go:356] "Generic (PLEG): container finished" podID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerID="3971bc50fade803150e154b6964a9879dc39a7ccbcc873b3f3b32f7650a7cfcd" exitCode=0
Sep 17 17:48:15.389673 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:15.389646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth" event={"ID":"14b1708e-46f9-4ff7-9429-866a62e9d8c9","Type":"ContainerDied","Data":"3971bc50fade803150e154b6964a9879dc39a7ccbcc873b3f3b32f7650a7cfcd"}
Sep 17 17:48:16.397814 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.397778    2580 generic.go:356] "Generic (PLEG): container finished" podID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerID="8108c795362d615181db54889e0fdca943262ab20eb3e60b7a76e8785ba30436" exitCode=0
Sep 17 17:48:16.398313 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.397866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc" event={"ID":"edc324ba-cf3e-4d87-8180-6e3b6c813ffe","Type":"ContainerDied","Data":"8108c795362d615181db54889e0fdca943262ab20eb3e60b7a76e8785ba30436"}
Sep 17 17:48:16.399772 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.399751    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerID="624fc3ba8924f4e399a01f57907f477d5e261e3813cd561195c125f589c83ddb" exitCode=0
Sep 17 17:48:16.399904 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.399802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g" event={"ID":"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a","Type":"ContainerDied","Data":"624fc3ba8924f4e399a01f57907f477d5e261e3813cd561195c125f589c83ddb"}
Sep 17 17:48:16.402353 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.402333    2580 generic.go:356] "Generic (PLEG): container finished" podID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerID="97f56632d820bb67b1ed81a0ba12ccf84943d1aafc996a2444f366118910876b" exitCode=0
Sep 17 17:48:16.402452 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.402417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth" event={"ID":"14b1708e-46f9-4ff7-9429-866a62e9d8c9","Type":"ContainerDied","Data":"97f56632d820bb67b1ed81a0ba12ccf84943d1aafc996a2444f366118910876b"}
Sep 17 17:48:16.536499 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.536467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:16.677906 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.677806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-util\") pod \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") "
Sep 17 17:48:16.677906 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.677857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ghmc\" (UniqueName: \"kubernetes.io/projected/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-kube-api-access-7ghmc\") pod \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") "
Sep 17 17:48:16.678186 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.677963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-bundle\") pod \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\" (UID: \"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc\") "
Sep 17 17:48:16.678522 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.678496    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-bundle" (OuterVolumeSpecName: "bundle") pod "b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" (UID: "b90f5ee0-7960-4fa5-9adc-14175ca1a8cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:16.680314 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.680280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-kube-api-access-7ghmc" (OuterVolumeSpecName: "kube-api-access-7ghmc") pod "b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" (UID: "b90f5ee0-7960-4fa5-9adc-14175ca1a8cc"). InnerVolumeSpecName "kube-api-access-7ghmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:16.683489 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.683452    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-util" (OuterVolumeSpecName: "util") pod "b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" (UID: "b90f5ee0-7960-4fa5-9adc-14175ca1a8cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:16.778884 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.778832    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:16.778884 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.778870    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-util\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:16.778884 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:16.778883    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ghmc\" (UniqueName: \"kubernetes.io/projected/b90f5ee0-7960-4fa5-9adc-14175ca1a8cc-kube-api-access-7ghmc\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:17.410510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.410480    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb"
Sep 17 17:48:17.411040 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.410478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75hgfb" event={"ID":"b90f5ee0-7960-4fa5-9adc-14175ca1a8cc","Type":"ContainerDied","Data":"0680b237a96a90aba489fb3fe02a65873646081af01068c3b5d3e2b1813567c9"}
Sep 17 17:48:17.411040 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.410612    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0680b237a96a90aba489fb3fe02a65873646081af01068c3b5d3e2b1813567c9"
Sep 17 17:48:17.412892 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.412852    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerID="40913607c21bc55d57db683e462829ba41bddb7c40a4d116972aa991accc760b" exitCode=0
Sep 17 17:48:17.413043 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.412933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g" event={"ID":"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a","Type":"ContainerDied","Data":"40913607c21bc55d57db683e462829ba41bddb7c40a4d116972aa991accc760b"}
Sep 17 17:48:17.554907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.554866    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:17.586448 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.586418    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:17.688110 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.687970    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-bundle\") pod \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") "
Sep 17 17:48:17.688110 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.688023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-bundle\") pod \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") "
Sep 17 17:48:17.688360 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.688115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-util\") pod \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") "
Sep 17 17:48:17.688360 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.688174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-util\") pod \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") "
Sep 17 17:48:17.688360 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.688203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x62s5\" (UniqueName: \"kubernetes.io/projected/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-kube-api-access-x62s5\") pod \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\" (UID: \"edc324ba-cf3e-4d87-8180-6e3b6c813ffe\") "
Sep 17 17:48:17.688360 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.688237    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb26v\" (UniqueName: \"kubernetes.io/projected/14b1708e-46f9-4ff7-9429-866a62e9d8c9-kube-api-access-bb26v\") pod \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\" (UID: \"14b1708e-46f9-4ff7-9429-866a62e9d8c9\") "
Sep 17 17:48:17.688674 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.688639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-bundle" (OuterVolumeSpecName: "bundle") pod "14b1708e-46f9-4ff7-9429-866a62e9d8c9" (UID: "14b1708e-46f9-4ff7-9429-866a62e9d8c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:17.688796 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.688744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-bundle" (OuterVolumeSpecName: "bundle") pod "edc324ba-cf3e-4d87-8180-6e3b6c813ffe" (UID: "edc324ba-cf3e-4d87-8180-6e3b6c813ffe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:17.690634 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.690601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b1708e-46f9-4ff7-9429-866a62e9d8c9-kube-api-access-bb26v" (OuterVolumeSpecName: "kube-api-access-bb26v") pod "14b1708e-46f9-4ff7-9429-866a62e9d8c9" (UID: "14b1708e-46f9-4ff7-9429-866a62e9d8c9"). InnerVolumeSpecName "kube-api-access-bb26v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:17.690763 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.690690    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-kube-api-access-x62s5" (OuterVolumeSpecName: "kube-api-access-x62s5") pod "edc324ba-cf3e-4d87-8180-6e3b6c813ffe" (UID: "edc324ba-cf3e-4d87-8180-6e3b6c813ffe"). InnerVolumeSpecName "kube-api-access-x62s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:17.693902 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.693856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-util" (OuterVolumeSpecName: "util") pod "edc324ba-cf3e-4d87-8180-6e3b6c813ffe" (UID: "edc324ba-cf3e-4d87-8180-6e3b6c813ffe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:17.694311 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.694288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-util" (OuterVolumeSpecName: "util") pod "14b1708e-46f9-4ff7-9429-866a62e9d8c9" (UID: "14b1708e-46f9-4ff7-9429-866a62e9d8c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:17.789464 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.789403    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:17.789464 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.789450    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:17.789464 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.789459    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b1708e-46f9-4ff7-9429-866a62e9d8c9-util\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:17.789464 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.789470    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-util\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:17.789464 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.789481    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62s5\" (UniqueName: \"kubernetes.io/projected/edc324ba-cf3e-4d87-8180-6e3b6c813ffe-kube-api-access-x62s5\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:17.789852 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:17.789494    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb26v\" (UniqueName: \"kubernetes.io/projected/14b1708e-46f9-4ff7-9429-866a62e9d8c9-kube-api-access-bb26v\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:18.420382 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.420338    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc"
Sep 17 17:48:18.420382 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.420352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctfpqc" event={"ID":"edc324ba-cf3e-4d87-8180-6e3b6c813ffe","Type":"ContainerDied","Data":"acb303576653b937d34d8a810f8f5850386b7e743b6dea468a7b53a168cf8758"}
Sep 17 17:48:18.420913 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.420415    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acb303576653b937d34d8a810f8f5850386b7e743b6dea468a7b53a168cf8758"
Sep 17 17:48:18.422631 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.422599    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth"
Sep 17 17:48:18.422793 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.422630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kvth" event={"ID":"14b1708e-46f9-4ff7-9429-866a62e9d8c9","Type":"ContainerDied","Data":"2a0a1b1675a416eedce2ad43f5693b0cdf46ea597d2f898f71a9f3fe8e8561af"}
Sep 17 17:48:18.422793 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.422693    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a0a1b1675a416eedce2ad43f5693b0cdf46ea597d2f898f71a9f3fe8e8561af"
Sep 17 17:48:18.564620 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.564582    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:18.696765 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.696667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-util\") pod \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") "
Sep 17 17:48:18.696765 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.696715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-bundle\") pod \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") "
Sep 17 17:48:18.696765 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.696769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg8vc\" (UniqueName: \"kubernetes.io/projected/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-kube-api-access-tg8vc\") pod \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\" (UID: \"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a\") "
Sep 17 17:48:18.697631 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.697588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-bundle" (OuterVolumeSpecName: "bundle") pod "7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" (UID: "7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:18.699116 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.699070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-kube-api-access-tg8vc" (OuterVolumeSpecName: "kube-api-access-tg8vc") pod "7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" (UID: "7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a"). InnerVolumeSpecName "kube-api-access-tg8vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:18.701493 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.701465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-util" (OuterVolumeSpecName: "util") pod "7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" (UID: "7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:18.797698 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.797638    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-util\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:18.797698 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.797689    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:18.797698 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:18.797700    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8vc\" (UniqueName: \"kubernetes.io/projected/7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a-kube-api-access-tg8vc\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:48:19.429015 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:19.428973    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g"
Sep 17 17:48:19.429015 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:19.428988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac4s8g" event={"ID":"7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a","Type":"ContainerDied","Data":"09ba5bc8af9fe8cd6b2bf66ca7ddf29007f7d4e33bcf9b2741d4db99d26d30ed"}
Sep 17 17:48:19.429502 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:19.429052    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09ba5bc8af9fe8cd6b2bf66ca7ddf29007f7d4e33bcf9b2741d4db99d26d30ed"
Sep 17 17:48:21.877938 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:21.877905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:22.668156 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:22.668117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:23.489224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:23.489187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:25.770852 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:25.770813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:25.867919 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:25.867876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:26.616878 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.616839    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"]
Sep 17 17:48:26.617197 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617183    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerName="util"
Sep 17 17:48:26.617243 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617199    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerName="util"
Sep 17 17:48:26.617243 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617212    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerName="util"
Sep 17 17:48:26.617243 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617217    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerName="util"
Sep 17 17:48:26.617243 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617225    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerName="pull"
Sep 17 17:48:26.617243 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617231    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerName="pull"
Sep 17 17:48:26.617243 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617241    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerName="util"
Sep 17 17:48:26.617243 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerName="util"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617253    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617279    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617286    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerName="pull"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617291    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerName="pull"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617299    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617304    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617310    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617317    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617329    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617335    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617342    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerName="util"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerName="util"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617355    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerName="pull"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617360    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerName="pull"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617366    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerName="pull"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617373    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerName="pull"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617435    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b90f5ee0-7960-4fa5-9adc-14175ca1a8cc" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617442    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b1708e-46f9-4ff7-9429-866a62e9d8c9" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617449    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc324ba-cf3e-4d87-8180-6e3b6c813ffe" containerName="extract"
Sep 17 17:48:26.617503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.617457    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7ff20f-9a22-4eaa-a8fd-c2766eceb88a" containerName="extract"
Sep 17 17:48:26.628342 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.628313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:26.631296 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.631267    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 17:48:26.631455 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.631369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vp4dt\""
Sep 17 17:48:26.631455 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.631377    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 17:48:26.637361 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.637337    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"]
Sep 17 17:48:26.772997 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.772952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq68\" (UniqueName: \"kubernetes.io/projected/b6cef6a2-dfb4-4399-9b2f-0ea208409579-kube-api-access-9rq68\") pod \"kuadrant-operator-controller-manager-6d4cb79c-rk659\" (UID: \"b6cef6a2-dfb4-4399-9b2f-0ea208409579\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:26.772997 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.773002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6cef6a2-dfb4-4399-9b2f-0ea208409579-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-rk659\" (UID: \"b6cef6a2-dfb4-4399-9b2f-0ea208409579\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:26.873679 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.873579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq68\" (UniqueName: \"kubernetes.io/projected/b6cef6a2-dfb4-4399-9b2f-0ea208409579-kube-api-access-9rq68\") pod \"kuadrant-operator-controller-manager-6d4cb79c-rk659\" (UID: \"b6cef6a2-dfb4-4399-9b2f-0ea208409579\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:26.873679 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.873623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6cef6a2-dfb4-4399-9b2f-0ea208409579-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-rk659\" (UID: \"b6cef6a2-dfb4-4399-9b2f-0ea208409579\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:26.873978 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.873962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6cef6a2-dfb4-4399-9b2f-0ea208409579-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-rk659\" (UID: \"b6cef6a2-dfb4-4399-9b2f-0ea208409579\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:26.882872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.882829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq68\" (UniqueName: \"kubernetes.io/projected/b6cef6a2-dfb4-4399-9b2f-0ea208409579-kube-api-access-9rq68\") pod \"kuadrant-operator-controller-manager-6d4cb79c-rk659\" (UID: \"b6cef6a2-dfb4-4399-9b2f-0ea208409579\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:26.940941 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:26.940893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:27.086224 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:27.086188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:27.228790 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:27.228760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"]
Sep 17 17:48:27.230612 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:48:27.230582    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cef6a2_dfb4_4399_9b2f_0ea208409579.slice/crio-da8518bee7e94bd401b4e7412a682938f525bcedb2a9266a1f071421b6997a30 WatchSource:0}: Error finding container da8518bee7e94bd401b4e7412a682938f525bcedb2a9266a1f071421b6997a30: Status 404 returned error can't find the container with id da8518bee7e94bd401b4e7412a682938f525bcedb2a9266a1f071421b6997a30
Sep 17 17:48:27.471915 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:27.471879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659" event={"ID":"b6cef6a2-dfb4-4399-9b2f-0ea208409579","Type":"ContainerStarted","Data":"da8518bee7e94bd401b4e7412a682938f525bcedb2a9266a1f071421b6997a30"}
Sep 17 17:48:28.133623 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:28.133582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:28.881755 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:28.881712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:29.291035 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:29.290996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:31.154976 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:31.154936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:31.256560 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:31.256522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:31.879651 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:31.879617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:32.963348 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:32.963310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:33.364701 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.364613    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"]
Sep 17 17:48:33.369563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.369543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"
Sep 17 17:48:33.372875 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.372854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xwg7f\""
Sep 17 17:48:33.389142 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.389113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"]
Sep 17 17:48:33.526316 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.526275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfh99\" (UniqueName: \"kubernetes.io/projected/c6db2741-11b5-4284-886b-a6829b728d09-kube-api-access-wfh99\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-k7bhn\" (UID: \"c6db2741-11b5-4284-886b-a6829b728d09\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"
Sep 17 17:48:33.627124 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.627003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfh99\" (UniqueName: \"kubernetes.io/projected/c6db2741-11b5-4284-886b-a6829b728d09-kube-api-access-wfh99\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-k7bhn\" (UID: \"c6db2741-11b5-4284-886b-a6829b728d09\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"
Sep 17 17:48:33.636827 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.636801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfh99\" (UniqueName: \"kubernetes.io/projected/c6db2741-11b5-4284-886b-a6829b728d09-kube-api-access-wfh99\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-k7bhn\" (UID: \"c6db2741-11b5-4284-886b-a6829b728d09\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"
Sep 17 17:48:33.680791 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.680737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"
Sep 17 17:48:33.962910 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:33.962872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"]
Sep 17 17:48:33.965847 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:48:33.965810    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6db2741_11b5_4284_886b_a6829b728d09.slice/crio-5f6692b0394b46932a64771b62ef181d4d35d8b68ce3f8896788f000ef46a7a5 WatchSource:0}: Error finding container 5f6692b0394b46932a64771b62ef181d4d35d8b68ce3f8896788f000ef46a7a5: Status 404 returned error can't find the container with id 5f6692b0394b46932a64771b62ef181d4d35d8b68ce3f8896788f000ef46a7a5
Sep 17 17:48:34.509890 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:34.509846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn" event={"ID":"c6db2741-11b5-4284-886b-a6829b728d09","Type":"ContainerStarted","Data":"5f6692b0394b46932a64771b62ef181d4d35d8b68ce3f8896788f000ef46a7a5"}
Sep 17 17:48:36.887192 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:36.887149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:36.958492 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:36.958438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:37.293726 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.293685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:37.394000 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.393967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:37.532984 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.532941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn" event={"ID":"c6db2741-11b5-4284-886b-a6829b728d09","Type":"ContainerStarted","Data":"7a7bac002377c287f88dc97228cec24e21885153357874466b0cb087edd3d744"}
Sep 17 17:48:37.533211 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.533187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"
Sep 17 17:48:37.535177 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.535143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659" event={"ID":"b6cef6a2-dfb4-4399-9b2f-0ea208409579","Type":"ContainerStarted","Data":"4babfa84655575319fe77e37c3db7fed1c0583507aecc2677e39ddf623622389"}
Sep 17 17:48:37.535344 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.535198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:37.553804 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.553688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn" podStartSLOduration=1.18268417 podStartE2EDuration="4.553669582s" totalImagesPullingTime="3.370985412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:37.551077358 +0000 UTC m=+493.890470951" watchObservedRunningTime="2026-09-17 17:48:37.553669582 +0000 UTC m=+493.893063174"
Sep 17 17:48:37.574744 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.574685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659" podStartSLOduration=1.454186243 podStartE2EDuration="11.574664843s" totalImagesPullingTime="10.1204786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:37.572727019 +0000 UTC m=+493.912120610" watchObservedRunningTime="2026-09-17 17:48:37.574664843 +0000 UTC m=+493.914058435"
Sep 17 17:48:37.687957 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.687918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:37.755497 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:37.755462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:40.085637 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:40.085597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:40.582016 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:40.581978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:43.365783 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:43.365736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:43.464952 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:43.464914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:48.543227 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:48.543191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-rk659"
Sep 17 17:48:48.543672 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:48.543286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-k7bhn"
Sep 17 17:48:50.023451 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:50.023408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:50.280734 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:50.280637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:50.424223 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:50.424184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:52.903532 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:52.903492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:53.019911 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:53.019866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:57.781396 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:57.781359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:57.879007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:57.878968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:58.104934 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:58.104828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:58.173895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:58.173853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:58.513801 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:58.513758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:58.592033 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:58.591999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:58.725949 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:58.725905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:58.827120 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:48:58.827004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:04.331545 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:04.331489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:05.707463 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:05.707422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:05.807939 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:05.807902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:11.524180 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:11.524074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:11.661526 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:11.661491    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:14.519661 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:14.519564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:14.616076 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:14.616040    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:19.958370 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:19.958323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:20.066952 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.066913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:20.144133 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.144099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:20.363484 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.363398    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"]
Sep 17 17:49:20.367015 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.366991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.369707 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.369680    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2gct4\""
Sep 17 17:49:20.369707 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.369701    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 17:49:20.379894 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.379866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"]
Sep 17 17:49:20.413246 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.413201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"]
Sep 17 17:49:20.441066 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.441030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njtgj\" (UniqueName: \"kubernetes.io/projected/16466b21-0f3d-49c6-abef-4d4c9c15a7ee-kube-api-access-njtgj\") pod \"limitador-limitador-84994f7ffc-wzhjg\" (UID: \"16466b21-0f3d-49c6-abef-4d4c9c15a7ee\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.441066 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.441067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16466b21-0f3d-49c6-abef-4d4c9c15a7ee-config-file\") pod \"limitador-limitador-84994f7ffc-wzhjg\" (UID: \"16466b21-0f3d-49c6-abef-4d4c9c15a7ee\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.541916 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.541875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njtgj\" (UniqueName: \"kubernetes.io/projected/16466b21-0f3d-49c6-abef-4d4c9c15a7ee-kube-api-access-njtgj\") pod \"limitador-limitador-84994f7ffc-wzhjg\" (UID: \"16466b21-0f3d-49c6-abef-4d4c9c15a7ee\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.541916 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.541914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16466b21-0f3d-49c6-abef-4d4c9c15a7ee-config-file\") pod \"limitador-limitador-84994f7ffc-wzhjg\" (UID: \"16466b21-0f3d-49c6-abef-4d4c9c15a7ee\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.542667 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.542635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16466b21-0f3d-49c6-abef-4d4c9c15a7ee-config-file\") pod \"limitador-limitador-84994f7ffc-wzhjg\" (UID: \"16466b21-0f3d-49c6-abef-4d4c9c15a7ee\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.551584 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.551553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njtgj\" (UniqueName: \"kubernetes.io/projected/16466b21-0f3d-49c6-abef-4d4c9c15a7ee-kube-api-access-njtgj\") pod \"limitador-limitador-84994f7ffc-wzhjg\" (UID: \"16466b21-0f3d-49c6-abef-4d4c9c15a7ee\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.678939 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.678900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:20.818617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.818573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:20.966340 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.966295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"]
Sep 17 17:49:20.969079 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:49:20.969041    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16466b21_0f3d_49c6_abef_4d4c9c15a7ee.slice/crio-8958e8660bfc53b67784f392b11d4b5efd2f787cdf774341284593ffa0538d5f WatchSource:0}: Error finding container 8958e8660bfc53b67784f392b11d4b5efd2f787cdf774341284593ffa0538d5f: Status 404 returned error can't find the container with id 8958e8660bfc53b67784f392b11d4b5efd2f787cdf774341284593ffa0538d5f
Sep 17 17:49:20.971572 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:20.971532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:21.010350 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.010310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-87z4r"]
Sep 17 17:49:21.015116 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.015071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-87z4r"
Sep 17 17:49:21.018221 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.018198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n7nvx\""
Sep 17 17:49:21.024385 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.024355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-87z4r"]
Sep 17 17:49:21.146619 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.146569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrx66\" (UniqueName: \"kubernetes.io/projected/0a488116-59f6-4bae-bef5-2c85f11448b9-kube-api-access-vrx66\") pod \"authorino-59c974bd48-87z4r\" (UID: \"0a488116-59f6-4bae-bef5-2c85f11448b9\") " pod="kuadrant-system/authorino-59c974bd48-87z4r"
Sep 17 17:49:21.247884 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.247778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrx66\" (UniqueName: \"kubernetes.io/projected/0a488116-59f6-4bae-bef5-2c85f11448b9-kube-api-access-vrx66\") pod \"authorino-59c974bd48-87z4r\" (UID: \"0a488116-59f6-4bae-bef5-2c85f11448b9\") " pod="kuadrant-system/authorino-59c974bd48-87z4r"
Sep 17 17:49:21.257128 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.257101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrx66\" (UniqueName: \"kubernetes.io/projected/0a488116-59f6-4bae-bef5-2c85f11448b9-kube-api-access-vrx66\") pod \"authorino-59c974bd48-87z4r\" (UID: \"0a488116-59f6-4bae-bef5-2c85f11448b9\") " pod="kuadrant-system/authorino-59c974bd48-87z4r"
Sep 17 17:49:21.336424 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.336376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-87z4r"
Sep 17 17:49:21.610518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.610424    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-87z4r"]
Sep 17 17:49:21.613715 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:49:21.613678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a488116_59f6_4bae_bef5_2c85f11448b9.slice/crio-920cfbfbaf7f7a9b666fa25265e5e5fe4567a4ea5d8a3662c47e6b8a44e06a07 WatchSource:0}: Error finding container 920cfbfbaf7f7a9b666fa25265e5e5fe4567a4ea5d8a3662c47e6b8a44e06a07: Status 404 returned error can't find the container with id 920cfbfbaf7f7a9b666fa25265e5e5fe4567a4ea5d8a3662c47e6b8a44e06a07
Sep 17 17:49:21.776245 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.776194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-87z4r" event={"ID":"0a488116-59f6-4bae-bef5-2c85f11448b9","Type":"ContainerStarted","Data":"920cfbfbaf7f7a9b666fa25265e5e5fe4567a4ea5d8a3662c47e6b8a44e06a07"}
Sep 17 17:49:21.777493 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:21.777467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg" event={"ID":"16466b21-0f3d-49c6-abef-4d4c9c15a7ee","Type":"ContainerStarted","Data":"8958e8660bfc53b67784f392b11d4b5efd2f787cdf774341284593ffa0538d5f"}
Sep 17 17:49:22.785021 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:22.784911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg" event={"ID":"16466b21-0f3d-49c6-abef-4d4c9c15a7ee","Type":"ContainerStarted","Data":"f28a4449bc3590410974d281309f2e6faa90745bf9829d60e567d9b3fcc8cec3"}
Sep 17 17:49:22.785468 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:22.785030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:22.808349 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:22.808289    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg" podStartSLOduration=1.291116161 podStartE2EDuration="2.808271048s" totalImagesPullingTime="1.517154887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:22.805223439 +0000 UTC m=+539.144617032" watchObservedRunningTime="2026-09-17 17:49:22.808271048 +0000 UTC m=+539.147664691"
Sep 17 17:49:24.798280 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:24.798231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-87z4r" event={"ID":"0a488116-59f6-4bae-bef5-2c85f11448b9","Type":"ContainerStarted","Data":"74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df"}
Sep 17 17:49:24.817165 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:24.817111    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-87z4r" podStartSLOduration=1.73047256 podStartE2EDuration="4.817074938s" totalImagesPullingTime="3.086602378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:24.816343675 +0000 UTC m=+541.155737281" watchObservedRunningTime="2026-09-17 17:49:24.817074938 +0000 UTC m=+541.156468529"
Sep 17 17:49:25.680029 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:25.679988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:33.792918 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:33.792877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wzhjg"
Sep 17 17:49:39.954557 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:39.954508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:40.226186 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:40.226058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:44.303386 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.303338    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-87z4r"]
Sep 17 17:49:44.303895 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.303577    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-87z4r" podUID="0a488116-59f6-4bae-bef5-2c85f11448b9" containerName="authorino" containerID="cri-o://74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df" gracePeriod=30
Sep 17 17:49:44.561575 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.561501    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-87z4r"
Sep 17 17:49:44.661353 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.661315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrx66\" (UniqueName: \"kubernetes.io/projected/0a488116-59f6-4bae-bef5-2c85f11448b9-kube-api-access-vrx66\") pod \"0a488116-59f6-4bae-bef5-2c85f11448b9\" (UID: \"0a488116-59f6-4bae-bef5-2c85f11448b9\") "
Sep 17 17:49:44.663673 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.663642    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a488116-59f6-4bae-bef5-2c85f11448b9-kube-api-access-vrx66" (OuterVolumeSpecName: "kube-api-access-vrx66") pod "0a488116-59f6-4bae-bef5-2c85f11448b9" (UID: "0a488116-59f6-4bae-bef5-2c85f11448b9"). InnerVolumeSpecName "kube-api-access-vrx66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:44.762098 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.762040    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrx66\" (UniqueName: \"kubernetes.io/projected/0a488116-59f6-4bae-bef5-2c85f11448b9-kube-api-access-vrx66\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:49:44.906443 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.906403    2580 generic.go:356] "Generic (PLEG): container finished" podID="0a488116-59f6-4bae-bef5-2c85f11448b9" containerID="74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df" exitCode=0
Sep 17 17:49:44.906650 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.906454    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-87z4r"
Sep 17 17:49:44.906650 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.906495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-87z4r" event={"ID":"0a488116-59f6-4bae-bef5-2c85f11448b9","Type":"ContainerDied","Data":"74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df"}
Sep 17 17:49:44.906650 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.906548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-87z4r" event={"ID":"0a488116-59f6-4bae-bef5-2c85f11448b9","Type":"ContainerDied","Data":"920cfbfbaf7f7a9b666fa25265e5e5fe4567a4ea5d8a3662c47e6b8a44e06a07"}
Sep 17 17:49:44.906650 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.906571    2580 scope.go:117] "RemoveContainer" containerID="74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df"
Sep 17 17:49:44.915650 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.915629    2580 scope.go:117] "RemoveContainer" containerID="74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df"
Sep 17 17:49:44.916010 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:49:44.915987    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df\": container with ID starting with 74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df not found: ID does not exist" containerID="74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df"
Sep 17 17:49:44.916052 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.916024    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df"} err="failed to get container status \"74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df\": rpc error: code = NotFound desc = could not find container \"74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df\": container with ID starting with 74fc1abd3c9bb32e89d07593c3538792423cf64eaf91933d41ba8570a00332df not found: ID does not exist"
Sep 17 17:49:44.927356 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.927324    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-87z4r"]
Sep 17 17:49:44.932072 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:44.932045    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-87z4r"]
Sep 17 17:49:46.046319 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:46.046281    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a488116-59f6-4bae-bef5-2c85f11448b9" path="/var/lib/kubelet/pods/0a488116-59f6-4bae-bef5-2c85f11448b9/volumes"
Sep 17 17:49:48.532437 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:48.532396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:48.632869 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:49:48.632829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:08.157749 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.157705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"]
Sep 17 17:50:08.158257 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.157941    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h" podUID="97f98e88-f419-4159-a365-83f338cf10f9" containerName="discovery" containerID="cri-o://2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5" gracePeriod=30
Sep 17 17:50:08.408440 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.408366    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:50:08.467038 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s75wr\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-kube-api-access-s75wr\") pod \"97f98e88-f419-4159-a365-83f338cf10f9\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") "
Sep 17 17:50:08.467248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-cacerts\") pod \"97f98e88-f419-4159-a365-83f338cf10f9\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") "
Sep 17 17:50:08.467248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-dns-cert\") pod \"97f98e88-f419-4159-a365-83f338cf10f9\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") "
Sep 17 17:50:08.467248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467123    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-kubeconfig\") pod \"97f98e88-f419-4159-a365-83f338cf10f9\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") "
Sep 17 17:50:08.467248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467159    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-ca-configmap\") pod \"97f98e88-f419-4159-a365-83f338cf10f9\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") "
Sep 17 17:50:08.467248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-istio-token\") pod \"97f98e88-f419-4159-a365-83f338cf10f9\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") "
Sep 17 17:50:08.467248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f98e88-f419-4159-a365-83f338cf10f9-local-certs\") pod \"97f98e88-f419-4159-a365-83f338cf10f9\" (UID: \"97f98e88-f419-4159-a365-83f338cf10f9\") "
Sep 17 17:50:08.467677 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.467650    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "97f98e88-f419-4159-a365-83f338cf10f9" (UID: "97f98e88-f419-4159-a365-83f338cf10f9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:08.470068 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.470007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "97f98e88-f419-4159-a365-83f338cf10f9" (UID: "97f98e88-f419-4159-a365-83f338cf10f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:08.470216 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.470115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-cacerts" (OuterVolumeSpecName: "cacerts") pod "97f98e88-f419-4159-a365-83f338cf10f9" (UID: "97f98e88-f419-4159-a365-83f338cf10f9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:08.470279 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.470246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "97f98e88-f419-4159-a365-83f338cf10f9" (UID: "97f98e88-f419-4159-a365-83f338cf10f9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:08.470279 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.470250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-kube-api-access-s75wr" (OuterVolumeSpecName: "kube-api-access-s75wr") pod "97f98e88-f419-4159-a365-83f338cf10f9" (UID: "97f98e88-f419-4159-a365-83f338cf10f9"). InnerVolumeSpecName "kube-api-access-s75wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:08.470279 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.470265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "97f98e88-f419-4159-a365-83f338cf10f9" (UID: "97f98e88-f419-4159-a365-83f338cf10f9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:08.470437 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.470342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f98e88-f419-4159-a365-83f338cf10f9-local-certs" (OuterVolumeSpecName: "local-certs") pod "97f98e88-f419-4159-a365-83f338cf10f9" (UID: "97f98e88-f419-4159-a365-83f338cf10f9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:08.568440 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.568402    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.568440 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.568431    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f98e88-f419-4159-a365-83f338cf10f9-local-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.568440 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.568441    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75wr\" (UniqueName: \"kubernetes.io/projected/97f98e88-f419-4159-a365-83f338cf10f9-kube-api-access-s75wr\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.568440 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.568451    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-cacerts\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.568722 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.568463    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-dns-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.568722 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.568473    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97f98e88-f419-4159-a365-83f338cf10f9-istio-kubeconfig\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.568722 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:08.568482    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97f98e88-f419-4159-a365-83f338cf10f9-istio-csr-ca-configmap\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:50:09.038292 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.038253    2580 generic.go:356] "Generic (PLEG): container finished" podID="97f98e88-f419-4159-a365-83f338cf10f9" containerID="2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5" exitCode=0
Sep 17 17:50:09.038525 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.038309    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"
Sep 17 17:50:09.038525 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.038352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h" event={"ID":"97f98e88-f419-4159-a365-83f338cf10f9","Type":"ContainerDied","Data":"2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5"}
Sep 17 17:50:09.038525 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.038444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h" event={"ID":"97f98e88-f419-4159-a365-83f338cf10f9","Type":"ContainerDied","Data":"368aac0fdae6173673453604f48b98867a39ab2d5dea6813513bae2be205cb8f"}
Sep 17 17:50:09.038525 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.038468    2580 scope.go:117] "RemoveContainer" containerID="2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5"
Sep 17 17:50:09.048188 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.048166    2580 scope.go:117] "RemoveContainer" containerID="2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5"
Sep 17 17:50:09.048515 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:50:09.048492    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5\": container with ID starting with 2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5 not found: ID does not exist" containerID="2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5"
Sep 17 17:50:09.048581 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.048524    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5"} err="failed to get container status \"2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5\": rpc error: code = NotFound desc = could not find container \"2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5\": container with ID starting with 2754900215fd74534ec18a700f97a1108b09950185c04fc29f8c7b9d620a86b5 not found: ID does not exist"
Sep 17 17:50:09.067093 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.067045    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"]
Sep 17 17:50:09.070852 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:09.070820    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l7z8h"]
Sep 17 17:50:10.046704 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:10.046669    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f98e88-f419-4159-a365-83f338cf10f9" path="/var/lib/kubelet/pods/97f98e88-f419-4159-a365-83f338cf10f9/volumes"
Sep 17 17:50:19.670642 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.670595    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:19.748522 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.748483    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-glvn4"]
Sep 17 17:50:19.748859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.748847    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f98e88-f419-4159-a365-83f338cf10f9" containerName="discovery"
Sep 17 17:50:19.748901 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.748861    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f98e88-f419-4159-a365-83f338cf10f9" containerName="discovery"
Sep 17 17:50:19.748901 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.748875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a488116-59f6-4bae-bef5-2c85f11448b9" containerName="authorino"
Sep 17 17:50:19.748901 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.748881    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a488116-59f6-4bae-bef5-2c85f11448b9" containerName="authorino"
Sep 17 17:50:19.748994 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.748948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f98e88-f419-4159-a365-83f338cf10f9" containerName="discovery"
Sep 17 17:50:19.748994 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.748959    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a488116-59f6-4bae-bef5-2c85f11448b9" containerName="authorino"
Sep 17 17:50:19.751944 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.751925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:19.754578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.754540    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:50:19.754578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.754559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-fvhm5\""
Sep 17 17:50:19.755058 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.754562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:50:19.755582 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.755563    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:50:19.762996 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.762966    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-glvn4"]
Sep 17 17:50:19.775031 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.774994    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7c5846b585-ckxfg"]
Sep 17 17:50:19.778673 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.778644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:19.781809 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.781767    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 17:50:19.782003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.781848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-tmblz\""
Sep 17 17:50:19.782077 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.782052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:19.790273 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.790241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7c5846b585-ckxfg"]
Sep 17 17:50:19.801911 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.801872    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-sgccl"]
Sep 17 17:50:19.807869 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.807838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:19.810475 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.810444    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:50:19.810654 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.810534    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-89gfc\""
Sep 17 17:50:19.816193 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.816163    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sgccl"]
Sep 17 17:50:19.859271 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.859222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwr47\" (UniqueName: \"kubernetes.io/projected/98cfa362-bc49-4abd-9574-8649756011c7-kube-api-access-vwr47\") pod \"llmisvc-controller-manager-7c5846b585-ckxfg\" (UID: \"98cfa362-bc49-4abd-9574-8649756011c7\") " pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:19.859271 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.859269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4v7\" (UniqueName: \"kubernetes.io/projected/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-kube-api-access-7z4v7\") pod \"kserve-controller-manager-5c5cdf4877-glvn4\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") " pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:19.859508 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.859337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cfa362-bc49-4abd-9574-8649756011c7-cert\") pod \"llmisvc-controller-manager-7c5846b585-ckxfg\" (UID: \"98cfa362-bc49-4abd-9574-8649756011c7\") " pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:19.859508 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.859422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-cert\") pod \"kserve-controller-manager-5c5cdf4877-glvn4\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") " pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:19.925074 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.924973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:19.960772 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.960734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-cert\") pod \"kserve-controller-manager-5c5cdf4877-glvn4\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") " pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:19.960964 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.960797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93140f91-8283-48a7-80b6-1399328a3123-data\") pod \"seaweedfs-64568bcd49-sgccl\" (UID: \"93140f91-8283-48a7-80b6-1399328a3123\") " pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:19.960964 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.960835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbw88\" (UniqueName: \"kubernetes.io/projected/93140f91-8283-48a7-80b6-1399328a3123-kube-api-access-wbw88\") pod \"seaweedfs-64568bcd49-sgccl\" (UID: \"93140f91-8283-48a7-80b6-1399328a3123\") " pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:19.960964 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.960871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwr47\" (UniqueName: \"kubernetes.io/projected/98cfa362-bc49-4abd-9574-8649756011c7-kube-api-access-vwr47\") pod \"llmisvc-controller-manager-7c5846b585-ckxfg\" (UID: \"98cfa362-bc49-4abd-9574-8649756011c7\") " pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:19.960964 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.960938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4v7\" (UniqueName: \"kubernetes.io/projected/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-kube-api-access-7z4v7\") pod \"kserve-controller-manager-5c5cdf4877-glvn4\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") " pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:19.961174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.961013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cfa362-bc49-4abd-9574-8649756011c7-cert\") pod \"llmisvc-controller-manager-7c5846b585-ckxfg\" (UID: \"98cfa362-bc49-4abd-9574-8649756011c7\") " pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:19.961174 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:50:19.961162    2580 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 17:50:19.961241 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:50:19.961230    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98cfa362-bc49-4abd-9574-8649756011c7-cert podName:98cfa362-bc49-4abd-9574-8649756011c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:50:20.461213663 +0000 UTC m=+596.800607237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/98cfa362-bc49-4abd-9574-8649756011c7-cert") pod "llmisvc-controller-manager-7c5846b585-ckxfg" (UID: "98cfa362-bc49-4abd-9574-8649756011c7") : secret "llmisvc-webhook-server-cert" not found
Sep 17 17:50:19.963617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.963589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-cert\") pod \"kserve-controller-manager-5c5cdf4877-glvn4\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") " pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:19.986717 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.986676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4v7\" (UniqueName: \"kubernetes.io/projected/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-kube-api-access-7z4v7\") pod \"kserve-controller-manager-5c5cdf4877-glvn4\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") " pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:19.986960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:19.986937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwr47\" (UniqueName: \"kubernetes.io/projected/98cfa362-bc49-4abd-9574-8649756011c7-kube-api-access-vwr47\") pod \"llmisvc-controller-manager-7c5846b585-ckxfg\" (UID: \"98cfa362-bc49-4abd-9574-8649756011c7\") " pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:20.061974 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.061928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93140f91-8283-48a7-80b6-1399328a3123-data\") pod \"seaweedfs-64568bcd49-sgccl\" (UID: \"93140f91-8283-48a7-80b6-1399328a3123\") " pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:20.061974 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.061980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbw88\" (UniqueName: \"kubernetes.io/projected/93140f91-8283-48a7-80b6-1399328a3123-kube-api-access-wbw88\") pod \"seaweedfs-64568bcd49-sgccl\" (UID: \"93140f91-8283-48a7-80b6-1399328a3123\") " pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:20.062467 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.062446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93140f91-8283-48a7-80b6-1399328a3123-data\") pod \"seaweedfs-64568bcd49-sgccl\" (UID: \"93140f91-8283-48a7-80b6-1399328a3123\") " pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:20.063762 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.063741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:20.096357 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.096327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbw88\" (UniqueName: \"kubernetes.io/projected/93140f91-8283-48a7-80b6-1399328a3123-kube-api-access-wbw88\") pod \"seaweedfs-64568bcd49-sgccl\" (UID: \"93140f91-8283-48a7-80b6-1399328a3123\") " pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:20.119562 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.119301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:20.163062 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.163005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:20.346960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.346929    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-glvn4"]
Sep 17 17:50:20.348862 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:50:20.348830    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335c7dfa_a745_47d2_9b7c_34ca1461ac9d.slice/crio-3c7f0624adc4885be186aea5530b093411a4ec76f193939113a84e06a33c422c WatchSource:0}: Error finding container 3c7f0624adc4885be186aea5530b093411a4ec76f193939113a84e06a33c422c: Status 404 returned error can't find the container with id 3c7f0624adc4885be186aea5530b093411a4ec76f193939113a84e06a33c422c
Sep 17 17:50:20.404640 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.404600    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sgccl"]
Sep 17 17:50:20.407997 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:50:20.407966    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93140f91_8283_48a7_80b6_1399328a3123.slice/crio-640e674e2c9b86f8005353d7ae46829586ca3bf1d064255e95bd52c1f009a5e0 WatchSource:0}: Error finding container 640e674e2c9b86f8005353d7ae46829586ca3bf1d064255e95bd52c1f009a5e0: Status 404 returned error can't find the container with id 640e674e2c9b86f8005353d7ae46829586ca3bf1d064255e95bd52c1f009a5e0
Sep 17 17:50:20.465223 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.465110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cfa362-bc49-4abd-9574-8649756011c7-cert\") pod \"llmisvc-controller-manager-7c5846b585-ckxfg\" (UID: \"98cfa362-bc49-4abd-9574-8649756011c7\") " pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:20.467589 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.467569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cfa362-bc49-4abd-9574-8649756011c7-cert\") pod \"llmisvc-controller-manager-7c5846b585-ckxfg\" (UID: \"98cfa362-bc49-4abd-9574-8649756011c7\") " pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:20.691659 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.691611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:20.971771 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:20.971730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7c5846b585-ckxfg"]
Sep 17 17:50:20.979233 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:50:20.979197    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98cfa362_bc49_4abd_9574_8649756011c7.slice/crio-29ddb96bfe0fb017834c0c86b1d3b10f15145b401e275f64f58193c249c60c4c WatchSource:0}: Error finding container 29ddb96bfe0fb017834c0c86b1d3b10f15145b401e275f64f58193c249c60c4c: Status 404 returned error can't find the container with id 29ddb96bfe0fb017834c0c86b1d3b10f15145b401e275f64f58193c249c60c4c
Sep 17 17:50:21.116331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:21.116228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sgccl" event={"ID":"93140f91-8283-48a7-80b6-1399328a3123","Type":"ContainerStarted","Data":"640e674e2c9b86f8005353d7ae46829586ca3bf1d064255e95bd52c1f009a5e0"}
Sep 17 17:50:21.118755 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:21.118713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg" event={"ID":"98cfa362-bc49-4abd-9574-8649756011c7","Type":"ContainerStarted","Data":"29ddb96bfe0fb017834c0c86b1d3b10f15145b401e275f64f58193c249c60c4c"}
Sep 17 17:50:21.120912 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:21.120876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4" event={"ID":"335c7dfa-a745-47d2-9b7c-34ca1461ac9d","Type":"ContainerStarted","Data":"3c7f0624adc4885be186aea5530b093411a4ec76f193939113a84e06a33c422c"}
Sep 17 17:50:21.200797 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:21.200736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:21.301973 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:21.301924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:25.024577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.024548    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:50:25.025214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.025189    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:50:25.056477 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.056454    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:50:25.155131 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.155069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4" event={"ID":"335c7dfa-a745-47d2-9b7c-34ca1461ac9d","Type":"ContainerStarted","Data":"65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148"}
Sep 17 17:50:25.155294 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.155170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:50:25.156981 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.156954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sgccl" event={"ID":"93140f91-8283-48a7-80b6-1399328a3123","Type":"ContainerStarted","Data":"b8347d8d91045ee75e70d5f9047716f43c5c303c4a58c45a04b355f42fc1a924"}
Sep 17 17:50:25.157139 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.157032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:25.158806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.158778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg" event={"ID":"98cfa362-bc49-4abd-9574-8649756011c7","Type":"ContainerStarted","Data":"4cbc3f5c6ddcaa9b29f3752e52114917266fe776dd32bb4be255bc8b391b0a87"}
Sep 17 17:50:25.159003 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.158985    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:25.178381 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.178308    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4" podStartSLOduration=1.699572742 podStartE2EDuration="6.178283581s" totalImagesPullingTime="4.478710839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:25.174505094 +0000 UTC m=+601.513898687" watchObservedRunningTime="2026-09-17 17:50:25.178283581 +0000 UTC m=+601.517677172"
Sep 17 17:50:25.193126 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.193034    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg" podStartSLOduration=2.179148104 podStartE2EDuration="6.193010183s" totalImagesPullingTime="4.013862079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:25.192962242 +0000 UTC m=+601.532355846" watchObservedRunningTime="2026-09-17 17:50:25.193010183 +0000 UTC m=+601.532403773"
Sep 17 17:50:25.214264 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:25.214205    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-sgccl" podStartSLOduration=1.570351049 podStartE2EDuration="6.214182145s" totalImagesPullingTime="4.643831096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:25.212334584 +0000 UTC m=+601.551728176" watchObservedRunningTime="2026-09-17 17:50:25.214182145 +0000 UTC m=+601.553575735"
Sep 17 17:50:31.167746 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:31.167712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-sgccl"
Sep 17 17:50:56.167262 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:56.167223    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7c5846b585-ckxfg"
Sep 17 17:50:56.172740 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:50:56.170424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:51:04.742670 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.742631    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-glvn4"]
Sep 17 17:51:04.743175 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.742854    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4" podUID="335c7dfa-a745-47d2-9b7c-34ca1461ac9d" containerName="manager" containerID="cri-o://65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148" gracePeriod=10
Sep 17 17:51:04.797642 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.797607    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-7zmcc"]
Sep 17 17:51:04.801006 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.800989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:04.828751 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.828715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-7zmcc"]
Sep 17 17:51:04.856528 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.856491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a1faa1-7a6b-43d3-8d10-5b571127690c-cert\") pod \"kserve-controller-manager-5c5cdf4877-7zmcc\" (UID: \"31a1faa1-7a6b-43d3-8d10-5b571127690c\") " pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:04.856683 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.856614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5stpm\" (UniqueName: \"kubernetes.io/projected/31a1faa1-7a6b-43d3-8d10-5b571127690c-kube-api-access-5stpm\") pod \"kserve-controller-manager-5c5cdf4877-7zmcc\" (UID: \"31a1faa1-7a6b-43d3-8d10-5b571127690c\") " pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:04.957680 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.957642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a1faa1-7a6b-43d3-8d10-5b571127690c-cert\") pod \"kserve-controller-manager-5c5cdf4877-7zmcc\" (UID: \"31a1faa1-7a6b-43d3-8d10-5b571127690c\") " pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:04.957846 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.957705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5stpm\" (UniqueName: \"kubernetes.io/projected/31a1faa1-7a6b-43d3-8d10-5b571127690c-kube-api-access-5stpm\") pod \"kserve-controller-manager-5c5cdf4877-7zmcc\" (UID: \"31a1faa1-7a6b-43d3-8d10-5b571127690c\") " pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:04.960111 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.960061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a1faa1-7a6b-43d3-8d10-5b571127690c-cert\") pod \"kserve-controller-manager-5c5cdf4877-7zmcc\" (UID: \"31a1faa1-7a6b-43d3-8d10-5b571127690c\") " pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:04.967404 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.967351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5stpm\" (UniqueName: \"kubernetes.io/projected/31a1faa1-7a6b-43d3-8d10-5b571127690c-kube-api-access-5stpm\") pod \"kserve-controller-manager-5c5cdf4877-7zmcc\" (UID: \"31a1faa1-7a6b-43d3-8d10-5b571127690c\") " pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:04.998554 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:04.998491    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:51:05.058171 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.058138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z4v7\" (UniqueName: \"kubernetes.io/projected/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-kube-api-access-7z4v7\") pod \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") "
Sep 17 17:51:05.058351 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.058211    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-cert\") pod \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\" (UID: \"335c7dfa-a745-47d2-9b7c-34ca1461ac9d\") "
Sep 17 17:51:05.060348 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.060320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-kube-api-access-7z4v7" (OuterVolumeSpecName: "kube-api-access-7z4v7") pod "335c7dfa-a745-47d2-9b7c-34ca1461ac9d" (UID: "335c7dfa-a745-47d2-9b7c-34ca1461ac9d"). InnerVolumeSpecName "kube-api-access-7z4v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:05.060473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.060384    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-cert" (OuterVolumeSpecName: "cert") pod "335c7dfa-a745-47d2-9b7c-34ca1461ac9d" (UID: "335c7dfa-a745-47d2-9b7c-34ca1461ac9d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:05.147296 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.147267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:05.159638 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.159602    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z4v7\" (UniqueName: \"kubernetes.io/projected/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-kube-api-access-7z4v7\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:51:05.159638 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.159639    2580 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335c7dfa-a745-47d2-9b7c-34ca1461ac9d-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:51:05.382397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.382285    2580 generic.go:356] "Generic (PLEG): container finished" podID="335c7dfa-a745-47d2-9b7c-34ca1461ac9d" containerID="65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148" exitCode=0
Sep 17 17:51:05.382397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.382333    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4"
Sep 17 17:51:05.382397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.382376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4" event={"ID":"335c7dfa-a745-47d2-9b7c-34ca1461ac9d","Type":"ContainerDied","Data":"65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148"}
Sep 17 17:51:05.382678 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.382428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-glvn4" event={"ID":"335c7dfa-a745-47d2-9b7c-34ca1461ac9d","Type":"ContainerDied","Data":"3c7f0624adc4885be186aea5530b093411a4ec76f193939113a84e06a33c422c"}
Sep 17 17:51:05.382678 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.382452    2580 scope.go:117] "RemoveContainer" containerID="65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148"
Sep 17 17:51:05.391921 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.391903    2580 scope.go:117] "RemoveContainer" containerID="65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148"
Sep 17 17:51:05.392239 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:51:05.392221    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148\": container with ID starting with 65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148 not found: ID does not exist" containerID="65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148"
Sep 17 17:51:05.392308 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.392251    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148"} err="failed to get container status \"65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148\": rpc error: code = NotFound desc = could not find container \"65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148\": container with ID starting with 65b5ab4ae5d424a732a89e10aa83250a71b321c903deb14cbf280d4f0f2c7148 not found: ID does not exist"
Sep 17 17:51:05.404573 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.404548    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-glvn4"]
Sep 17 17:51:05.411211 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.411184    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-glvn4"]
Sep 17 17:51:05.429321 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.429293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-7zmcc"]
Sep 17 17:51:05.431198 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:51:05.431175    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a1faa1_7a6b_43d3_8d10_5b571127690c.slice/crio-c03e53c89e6fa62e3f0352b2d5f2413e9d61897e54e15cd5ba6e00e121ad28f7 WatchSource:0}: Error finding container c03e53c89e6fa62e3f0352b2d5f2413e9d61897e54e15cd5ba6e00e121ad28f7: Status 404 returned error can't find the container with id c03e53c89e6fa62e3f0352b2d5f2413e9d61897e54e15cd5ba6e00e121ad28f7
Sep 17 17:51:05.432574 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:05.432556    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:51:06.046147 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:06.046062    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335c7dfa-a745-47d2-9b7c-34ca1461ac9d" path="/var/lib/kubelet/pods/335c7dfa-a745-47d2-9b7c-34ca1461ac9d/volumes"
Sep 17 17:51:06.389387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:06.389282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc" event={"ID":"31a1faa1-7a6b-43d3-8d10-5b571127690c","Type":"ContainerStarted","Data":"05885de385ce204dd8ac0ee6a5fd90a81b3fd184453ca2d82aa7e38e8be2113e"}
Sep 17 17:51:06.389387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:06.389342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc" event={"ID":"31a1faa1-7a6b-43d3-8d10-5b571127690c","Type":"ContainerStarted","Data":"c03e53c89e6fa62e3f0352b2d5f2413e9d61897e54e15cd5ba6e00e121ad28f7"}
Sep 17 17:51:06.389387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:06.389371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:06.424646 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:06.424586    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc" podStartSLOduration=1.938494971 podStartE2EDuration="2.424567394s" totalImagesPullingTime="486.072423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:06.423032668 +0000 UTC m=+642.762426259" watchObservedRunningTime="2026-09-17 17:51:06.424567394 +0000 UTC m=+642.763960985"
Sep 17 17:51:12.138267 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:12.138233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:33.400571 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:33.400536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:37.399370 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:37.399341    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-7zmcc"
Sep 17 17:51:39.113371 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.113340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:39.174934 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.174906    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-l559l"]
Sep 17 17:51:39.175335 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.175320    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335c7dfa-a745-47d2-9b7c-34ca1461ac9d" containerName="manager"
Sep 17 17:51:39.175392 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.175338    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="335c7dfa-a745-47d2-9b7c-34ca1461ac9d" containerName="manager"
Sep 17 17:51:39.175431 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.175405    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="335c7dfa-a745-47d2-9b7c-34ca1461ac9d" containerName="manager"
Sep 17 17:51:39.178736 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.178714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.186311 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.186294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 17:51:39.186584 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.186566    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-44ldf\""
Sep 17 17:51:39.196453 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.196422    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-8b9rk"]
Sep 17 17:51:39.199709 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.199690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.203668 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.203649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 17:51:39.203759 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.203708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-cw6vp\""
Sep 17 17:51:39.203859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.203843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 17:51:39.209056 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.209035    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-l559l"]
Sep 17 17:51:39.235888 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.235861    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-8b9rk"]
Sep 17 17:51:39.342147 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.342108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:39.347215 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.347191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vksmc\" (UniqueName: \"kubernetes.io/projected/2b610af7-cb50-4b55-9e6f-faf5e01adc31-kube-api-access-vksmc\") pod \"model-serving-api-7b954ffc95-l559l\" (UID: \"2b610af7-cb50-4b55-9e6f-faf5e01adc31\") " pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.347288 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.347244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f96617e-b442-43c2-aed7-b1cb5482b061-cert\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.347386 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.347295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmbh\" (UniqueName: \"kubernetes.io/projected/6f96617e-b442-43c2-aed7-b1cb5482b061-kube-api-access-8rmbh\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.347386 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.347341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f96617e-b442-43c2-aed7-b1cb5482b061-metrics-certs\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.347386 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.347360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b610af7-cb50-4b55-9e6f-faf5e01adc31-tls-certs\") pod \"model-serving-api-7b954ffc95-l559l\" (UID: \"2b610af7-cb50-4b55-9e6f-faf5e01adc31\") " pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.448359 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.448322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmbh\" (UniqueName: \"kubernetes.io/projected/6f96617e-b442-43c2-aed7-b1cb5482b061-kube-api-access-8rmbh\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.448563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.448382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f96617e-b442-43c2-aed7-b1cb5482b061-metrics-certs\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.448563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.448405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b610af7-cb50-4b55-9e6f-faf5e01adc31-tls-certs\") pod \"model-serving-api-7b954ffc95-l559l\" (UID: \"2b610af7-cb50-4b55-9e6f-faf5e01adc31\") " pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.448563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.448439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vksmc\" (UniqueName: \"kubernetes.io/projected/2b610af7-cb50-4b55-9e6f-faf5e01adc31-kube-api-access-vksmc\") pod \"model-serving-api-7b954ffc95-l559l\" (UID: \"2b610af7-cb50-4b55-9e6f-faf5e01adc31\") " pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.448563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.448476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f96617e-b442-43c2-aed7-b1cb5482b061-cert\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.450987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.450953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f96617e-b442-43c2-aed7-b1cb5482b061-cert\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.450987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.450965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b610af7-cb50-4b55-9e6f-faf5e01adc31-tls-certs\") pod \"model-serving-api-7b954ffc95-l559l\" (UID: \"2b610af7-cb50-4b55-9e6f-faf5e01adc31\") " pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.451163 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.451004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f96617e-b442-43c2-aed7-b1cb5482b061-metrics-certs\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.464036 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.464013    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vksmc\" (UniqueName: \"kubernetes.io/projected/2b610af7-cb50-4b55-9e6f-faf5e01adc31-kube-api-access-vksmc\") pod \"model-serving-api-7b954ffc95-l559l\" (UID: \"2b610af7-cb50-4b55-9e6f-faf5e01adc31\") " pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.464826 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.464807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmbh\" (UniqueName: \"kubernetes.io/projected/6f96617e-b442-43c2-aed7-b1cb5482b061-kube-api-access-8rmbh\") pod \"odh-model-controller-7b966d7984-8b9rk\" (UID: \"6f96617e-b442-43c2-aed7-b1cb5482b061\") " pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.469294 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.469271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:39.488108 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.488073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:39.510274 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.510249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:39.561367 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.561336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:39.771548 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.771522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-l559l"]
Sep 17 17:51:39.773542 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:51:39.773513    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b610af7_cb50_4b55_9e6f_faf5e01adc31.slice/crio-1b4b0d0481e83752d524d602c57d6b5b40eb3121d63bdb56e59a7b1e944a7095 WatchSource:0}: Error finding container 1b4b0d0481e83752d524d602c57d6b5b40eb3121d63bdb56e59a7b1e944a7095: Status 404 returned error can't find the container with id 1b4b0d0481e83752d524d602c57d6b5b40eb3121d63bdb56e59a7b1e944a7095
Sep 17 17:51:39.823486 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:39.823462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-8b9rk"]
Sep 17 17:51:39.825231 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:51:39.825202    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f96617e_b442_43c2_aed7_b1cb5482b061.slice/crio-253daa5fb18c7b07dbecb633245c3ebc1616f02d37f7f4c6099f1b28c68cadc0 WatchSource:0}: Error finding container 253daa5fb18c7b07dbecb633245c3ebc1616f02d37f7f4c6099f1b28c68cadc0: Status 404 returned error can't find the container with id 253daa5fb18c7b07dbecb633245c3ebc1616f02d37f7f4c6099f1b28c68cadc0
Sep 17 17:51:40.576135 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:40.576030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-8b9rk" event={"ID":"6f96617e-b442-43c2-aed7-b1cb5482b061","Type":"ContainerStarted","Data":"253daa5fb18c7b07dbecb633245c3ebc1616f02d37f7f4c6099f1b28c68cadc0"}
Sep 17 17:51:40.577823 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:40.577792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-l559l" event={"ID":"2b610af7-cb50-4b55-9e6f-faf5e01adc31","Type":"ContainerStarted","Data":"1b4b0d0481e83752d524d602c57d6b5b40eb3121d63bdb56e59a7b1e944a7095"}
Sep 17 17:51:40.850678 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:40.850534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:40.948329 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:40.948266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:43.597358 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:43.597323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-l559l" event={"ID":"2b610af7-cb50-4b55-9e6f-faf5e01adc31","Type":"ContainerStarted","Data":"8463f1340765e9a70ec273f54b1a70b213a3f42f8da4f0a349ad8997c62b9fa7"}
Sep 17 17:51:43.597826 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:43.597383    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:43.599017 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:43.598969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-8b9rk" event={"ID":"6f96617e-b442-43c2-aed7-b1cb5482b061","Type":"ContainerStarted","Data":"d76850feb2747791885b953e9015611763a7e3e0bd041e4741d336e39427b269"}
Sep 17 17:51:43.599179 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:43.599044    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:43.617684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:43.617640    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-l559l" podStartSLOduration=1.017007535 podStartE2EDuration="4.617625694s" totalImagesPullingTime="3.600618159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:43.616307041 +0000 UTC m=+679.955700633" watchObservedRunningTime="2026-09-17 17:51:43.617625694 +0000 UTC m=+679.957019284"
Sep 17 17:51:43.634350 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:43.634271    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-8b9rk" podStartSLOduration=1.035855891 podStartE2EDuration="4.634257453s" totalImagesPullingTime="3.598401562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:43.633820406 +0000 UTC m=+679.973213998" watchObservedRunningTime="2026-09-17 17:51:43.634257453 +0000 UTC m=+679.973651044"
Sep 17 17:51:54.607492 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:54.607456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-8b9rk"
Sep 17 17:51:54.609048 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:54.609027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-l559l"
Sep 17 17:51:55.433177 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.433132    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-ms4rf"]
Sep 17 17:51:55.436839 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.436816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ms4rf"
Sep 17 17:51:55.439480 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.439451    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:51:55.447511 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.447480    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ms4rf"]
Sep 17 17:51:55.482412 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.482375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2292953-2b2b-4463-bba2-dff327fe7cd9-model-cache\") pod \"s3-init-ms4rf\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") " pod="kserve/s3-init-ms4rf"
Sep 17 17:51:55.482614 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.482454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2w4c\" (UniqueName: \"kubernetes.io/projected/f2292953-2b2b-4463-bba2-dff327fe7cd9-kube-api-access-d2w4c\") pod \"s3-init-ms4rf\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") " pod="kserve/s3-init-ms4rf"
Sep 17 17:51:55.583510 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.583478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2292953-2b2b-4463-bba2-dff327fe7cd9-model-cache\") pod \"s3-init-ms4rf\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") " pod="kserve/s3-init-ms4rf"
Sep 17 17:51:55.583689 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.583565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2w4c\" (UniqueName: \"kubernetes.io/projected/f2292953-2b2b-4463-bba2-dff327fe7cd9-kube-api-access-d2w4c\") pod \"s3-init-ms4rf\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") " pod="kserve/s3-init-ms4rf"
Sep 17 17:51:55.583872 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.583851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2292953-2b2b-4463-bba2-dff327fe7cd9-model-cache\") pod \"s3-init-ms4rf\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") " pod="kserve/s3-init-ms4rf"
Sep 17 17:51:55.592033 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.592001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2w4c\" (UniqueName: \"kubernetes.io/projected/f2292953-2b2b-4463-bba2-dff327fe7cd9-kube-api-access-d2w4c\") pod \"s3-init-ms4rf\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") " pod="kserve/s3-init-ms4rf"
Sep 17 17:51:55.765357 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:55.765267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ms4rf"
Sep 17 17:51:56.044943 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:51:56.044911    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2292953_2b2b_4463_bba2_dff327fe7cd9.slice/crio-ead59691bbcd0c02a736a0e769d014113aa4a6f8a1fd93046905738f55e42241 WatchSource:0}: Error finding container ead59691bbcd0c02a736a0e769d014113aa4a6f8a1fd93046905738f55e42241: Status 404 returned error can't find the container with id ead59691bbcd0c02a736a0e769d014113aa4a6f8a1fd93046905738f55e42241
Sep 17 17:51:56.048139 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:56.048118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ms4rf"]
Sep 17 17:51:56.673198 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:56.673158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ms4rf" event={"ID":"f2292953-2b2b-4463-bba2-dff327fe7cd9","Type":"ContainerStarted","Data":"ead59691bbcd0c02a736a0e769d014113aa4a6f8a1fd93046905738f55e42241"}
Sep 17 17:51:59.692616 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:51:59.692570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ms4rf" event={"ID":"f2292953-2b2b-4463-bba2-dff327fe7cd9","Type":"ContainerStarted","Data":"29cfcc214198a5d23b50706a3d04a371afb34571269bbcefaa021f66b1148bdc"}
Sep 17 17:52:01.705934 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:52:01.705901    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2292953-2b2b-4463-bba2-dff327fe7cd9" containerID="29cfcc214198a5d23b50706a3d04a371afb34571269bbcefaa021f66b1148bdc" exitCode=0
Sep 17 17:52:01.706387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:52:01.705969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ms4rf" event={"ID":"f2292953-2b2b-4463-bba2-dff327fe7cd9","Type":"ContainerDied","Data":"29cfcc214198a5d23b50706a3d04a371afb34571269bbcefaa021f66b1148bdc"}
Sep 17 17:52:06.740500 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:52:06.740467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ms4rf" event={"ID":"f2292953-2b2b-4463-bba2-dff327fe7cd9","Type":"ContainerStarted","Data":"f1272133887778d5ae8d1fefb63680f345c6aa600524b1c93d1a2c2fc29afaa5"}
Sep 17 17:52:06.762057 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:52:06.762007    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-ms4rf" podStartSLOduration=2.279527752 podStartE2EDuration="11.761992272s" totalImagesPullingTime="7.766613059s" totalInitContainerRuntime="1.715851461s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:06.760656675 +0000 UTC m=+703.100050265" watchObservedRunningTime="2026-09-17 17:52:06.761992272 +0000 UTC m=+703.101385863"
Sep 17 17:52:29.346759 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:52:29.346672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:29.448328 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:52:29.448298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:39.214613 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:52:39.214576    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:03.432954 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:03.432925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:04.407220 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:04.407178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:04.482075 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:04.482038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:12.092569 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:12.092535    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2292953-2b2b-4463-bba2-dff327fe7cd9" containerID="f1272133887778d5ae8d1fefb63680f345c6aa600524b1c93d1a2c2fc29afaa5" exitCode=0
Sep 17 17:53:12.092993 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:12.092591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ms4rf" event={"ID":"f2292953-2b2b-4463-bba2-dff327fe7cd9","Type":"ContainerDied","Data":"f1272133887778d5ae8d1fefb63680f345c6aa600524b1c93d1a2c2fc29afaa5"}
Sep 17 17:53:13.233784 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:13.233760    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ms4rf"
Sep 17 17:53:13.376429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:13.376321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2w4c\" (UniqueName: \"kubernetes.io/projected/f2292953-2b2b-4463-bba2-dff327fe7cd9-kube-api-access-d2w4c\") pod \"f2292953-2b2b-4463-bba2-dff327fe7cd9\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") "
Sep 17 17:53:13.376429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:13.376385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2292953-2b2b-4463-bba2-dff327fe7cd9-model-cache\") pod \"f2292953-2b2b-4463-bba2-dff327fe7cd9\" (UID: \"f2292953-2b2b-4463-bba2-dff327fe7cd9\") "
Sep 17 17:53:13.378572 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:13.378544    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2292953-2b2b-4463-bba2-dff327fe7cd9-kube-api-access-d2w4c" (OuterVolumeSpecName: "kube-api-access-d2w4c") pod "f2292953-2b2b-4463-bba2-dff327fe7cd9" (UID: "f2292953-2b2b-4463-bba2-dff327fe7cd9"). InnerVolumeSpecName "kube-api-access-d2w4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:13.395563 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:13.395535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2292953-2b2b-4463-bba2-dff327fe7cd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2292953-2b2b-4463-bba2-dff327fe7cd9" (UID: "f2292953-2b2b-4463-bba2-dff327fe7cd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:13.477123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:13.477067    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2w4c\" (UniqueName: \"kubernetes.io/projected/f2292953-2b2b-4463-bba2-dff327fe7cd9-kube-api-access-d2w4c\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:53:13.477123 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:13.477118    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2292953-2b2b-4463-bba2-dff327fe7cd9-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:53:14.104565 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:14.104539    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ms4rf"
Sep 17 17:53:14.104565 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:14.104550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ms4rf" event={"ID":"f2292953-2b2b-4463-bba2-dff327fe7cd9","Type":"ContainerDied","Data":"ead59691bbcd0c02a736a0e769d014113aa4a6f8a1fd93046905738f55e42241"}
Sep 17 17:53:14.104757 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:14.104586    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ead59691bbcd0c02a736a0e769d014113aa4a6f8a1fd93046905738f55e42241"
Sep 17 17:53:14.477617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:14.477579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:14.578297 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:14.578262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:21.583412 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:21.583366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:26.638395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:26.638356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:26.741436 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:26.741396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:32.458020 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.457984    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"]
Sep 17 17:53:32.458531 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.458516    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2292953-2b2b-4463-bba2-dff327fe7cd9" containerName="download-hf-model"
Sep 17 17:53:32.458578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.458535    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2292953-2b2b-4463-bba2-dff327fe7cd9" containerName="download-hf-model"
Sep 17 17:53:32.458578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.458558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2292953-2b2b-4463-bba2-dff327fe7cd9" containerName="s3-init"
Sep 17 17:53:32.458578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.458566    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2292953-2b2b-4463-bba2-dff327fe7cd9" containerName="s3-init"
Sep 17 17:53:32.458670 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.458661    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2292953-2b2b-4463-bba2-dff327fe7cd9" containerName="s3-init"
Sep 17 17:53:32.462071 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.462051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.467730 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.467703    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-gk487\""
Sep 17 17:53:32.467946 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.467933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 17:53:32.468844 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.468820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 17:53:32.469060 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.469038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 17:53:32.479233 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.479206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"]
Sep 17 17:53:32.483023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.482998    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:32.538159 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.538118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzpt\" (UniqueName: \"kubernetes.io/projected/07c9d8a1-5319-43f0-80a2-4af08d2022f2-kube-api-access-2vzpt\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.538323 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.538173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.538323 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.538200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.538323 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.538308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.538450 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.538336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.538450 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.538371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.638976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639288 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639288 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzpt\" (UniqueName: \"kubernetes.io/projected/07c9d8a1-5319-43f0-80a2-4af08d2022f2-kube-api-access-2vzpt\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639288 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639288 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639715 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639715 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.639715 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.639559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.641512 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.641486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.641773 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.641753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.648167 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.648147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzpt\" (UniqueName: \"kubernetes.io/projected/07c9d8a1-5319-43f0-80a2-4af08d2022f2-kube-api-access-2vzpt\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2pns7\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.772264 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.772181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:32.833602 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.833565    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"]
Sep 17 17:53:32.839138 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.839066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:32.842109 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.841892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-6bkd7\""
Sep 17 17:53:32.855283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.855135    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:32.857023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.856995    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"]
Sep 17 17:53:32.942265 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.942228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7xkx\" (UniqueName: \"kubernetes.io/projected/cc7de744-5d30-4a2e-8744-55ead34930d3-kube-api-access-l7xkx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:32.942439 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:32.942272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7de744-5d30-4a2e-8744-55ead34930d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:33.043399 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.043312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xkx\" (UniqueName: \"kubernetes.io/projected/cc7de744-5d30-4a2e-8744-55ead34930d3-kube-api-access-l7xkx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:33.043399 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.043367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7de744-5d30-4a2e-8744-55ead34930d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:33.045869 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.045838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7de744-5d30-4a2e-8744-55ead34930d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:33.057378 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.057348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"]
Sep 17 17:53:33.058229 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.058210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xkx\" (UniqueName: \"kubernetes.io/projected/cc7de744-5d30-4a2e-8744-55ead34930d3-kube-api-access-l7xkx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:33.059211 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:53:33.059188    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c9d8a1_5319_43f0_80a2_4af08d2022f2.slice/crio-1ac8ce96554136dfd186e848820202297fba21eea72bf1666a04d674f39be3a2 WatchSource:0}: Error finding container 1ac8ce96554136dfd186e848820202297fba21eea72bf1666a04d674f39be3a2: Status 404 returned error can't find the container with id 1ac8ce96554136dfd186e848820202297fba21eea72bf1666a04d674f39be3a2
Sep 17 17:53:33.161109 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.161044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:33.206098 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.206052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" event={"ID":"07c9d8a1-5319-43f0-80a2-4af08d2022f2","Type":"ContainerStarted","Data":"1ac8ce96554136dfd186e848820202297fba21eea72bf1666a04d674f39be3a2"}
Sep 17 17:53:33.443806 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:33.443774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"]
Sep 17 17:53:33.487809 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:53:33.487766    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7de744_5d30_4a2e_8744_55ead34930d3.slice/crio-276c940f8ca5d5849458ebceb3a731c0d25c3149404f85680ebe6eff72666867 WatchSource:0}: Error finding container 276c940f8ca5d5849458ebceb3a731c0d25c3149404f85680ebe6eff72666867: Status 404 returned error can't find the container with id 276c940f8ca5d5849458ebceb3a731c0d25c3149404f85680ebe6eff72666867
Sep 17 17:53:34.215505 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:34.215454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" event={"ID":"cc7de744-5d30-4a2e-8744-55ead34930d3","Type":"ContainerStarted","Data":"276c940f8ca5d5849458ebceb3a731c0d25c3149404f85680ebe6eff72666867"}
Sep 17 17:53:35.225721 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:35.225686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" event={"ID":"07c9d8a1-5319-43f0-80a2-4af08d2022f2","Type":"ContainerStarted","Data":"fd2ba366537ba0018a1dfdb4b6a2da2ef7cd69292a441710b9671a21520ddb69"}
Sep 17 17:53:35.254508 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:35.254451    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" podStartSLOduration=1.559409671 podStartE2EDuration="3.254428373s" totalImagesPullingTime="1.695018702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:35.251401077 +0000 UTC m=+791.590794669" watchObservedRunningTime="2026-09-17 17:53:35.254428373 +0000 UTC m=+791.593821991"
Sep 17 17:53:35.773026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:35.772929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:36.235302 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:36.235259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" event={"ID":"cc7de744-5d30-4a2e-8744-55ead34930d3","Type":"ContainerStarted","Data":"0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940"}
Sep 17 17:53:36.257297 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:36.257241    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" podStartSLOduration=1.769362568 podStartE2EDuration="4.2572234s" totalImagesPullingTime="2.487860832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:36.254900886 +0000 UTC m=+792.594294479" watchObservedRunningTime="2026-09-17 17:53:36.2572234 +0000 UTC m=+792.596616990"
Sep 17 17:53:37.241490 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:37.241460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:53:42.772589 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:42.772548    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:42.784814 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:42.784794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:43.284006 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:43.283979    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:53:55.366817 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:55.366776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:58.401497 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.401463    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"]
Sep 17 17:53:58.405167 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.405149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.408679 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.408644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:58.409075 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.409056    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-h9zzl\""
Sep 17 17:53:58.409075 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.409064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 17:53:58.409250 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.409149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 17:53:58.409250 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.409174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 17:53:58.417310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.417288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"]
Sep 17 17:53:58.470196 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.470162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.470395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.470231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.470395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.470294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwlj\" (UniqueName: \"kubernetes.io/projected/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-kube-api-access-hhwlj\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.470395 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.470330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.470561 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.470411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.470561 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.470448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-home\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.570878 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.570839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwlj\" (UniqueName: \"kubernetes.io/projected/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-kube-api-access-hhwlj\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.570878 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.570882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.571158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.570911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.571158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.570926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-home\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.571158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.570960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.571158 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.571013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.571381 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.571354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.571425 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.571384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-home\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.571425 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.571410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.573312 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.573279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.573499 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.573484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.579449 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.579423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwlj\" (UniqueName: \"kubernetes.io/projected/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-kube-api-access-hhwlj\") pod \"fc-smoke-test-kserve-7f4687897-7bghl\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.669297 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.669210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:58.716023 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.715995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:53:58.991335 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:58.991313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"]
Sep 17 17:53:58.994165 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:53:58.994139    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3b9fb8_3517_49f2_84bb_8eb9923b31a5.slice/crio-69945953923c0fe7fdcc26f0942d3140dbf299e3e9dde45136567f2a6c3e910e WatchSource:0}: Error finding container 69945953923c0fe7fdcc26f0942d3140dbf299e3e9dde45136567f2a6c3e910e: Status 404 returned error can't find the container with id 69945953923c0fe7fdcc26f0942d3140dbf299e3e9dde45136567f2a6c3e910e
Sep 17 17:53:59.363572 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:59.363477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" event={"ID":"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5","Type":"ContainerStarted","Data":"28151c11c6b3fa09f92717eeb70866b71afbc6c725c4fd0c0e7e1b478c0fd2de"}
Sep 17 17:53:59.363572 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:59.363532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" event={"ID":"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5","Type":"ContainerStarted","Data":"69945953923c0fe7fdcc26f0942d3140dbf299e3e9dde45136567f2a6c3e910e"}
Sep 17 17:53:59.382976 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:59.382922    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" podStartSLOduration=1.382906448 podStartE2EDuration="1.382906448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:59.379854154 +0000 UTC m=+815.719247747" watchObservedRunningTime="2026-09-17 17:53:59.382906448 +0000 UTC m=+815.722300038"
Sep 17 17:53:59.716731 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:53:59.716699    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:54:08.248126 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:08.248077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:54:08.716968 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:08.716935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:54:08.733241 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:08.733212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:54:09.429238 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:09.429208    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:54:12.861312 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:12.861274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:17.185432 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.185389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:17.192352 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.192321    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"]
Sep 17 17:54:17.202159 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:54:17.202120    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:54:17.202159 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:17.202159 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:17.202159 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="fd2ba366537ba0018a1dfdb4b6a2da2ef7cd69292a441710b9671a21520ddb69" cmd=["/bin/sleep","15"]
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.202199    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" message="" err=<
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:54:17.202216    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" podUID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" containerName="main" containerID="cri-o://fd2ba366537ba0018a1dfdb4b6a2da2ef7cd69292a441710b9671a21520ddb69"
Sep 17 17:54:17.202458 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.202276    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" podUID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" containerName="main" containerID="cri-o://fd2ba366537ba0018a1dfdb4b6a2da2ef7cd69292a441710b9671a21520ddb69" gracePeriod=60
Sep 17 17:54:17.239299 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.239267    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"]
Sep 17 17:54:17.272775 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.272742    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" podUID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 17:54:17.364627 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.364585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:17.471721 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.471627    2580 generic.go:356] "Generic (PLEG): container finished" podID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" containerID="fd2ba366537ba0018a1dfdb4b6a2da2ef7cd69292a441710b9671a21520ddb69" exitCode=0
Sep 17 17:54:17.471721 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.471706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" event={"ID":"07c9d8a1-5319-43f0-80a2-4af08d2022f2","Type":"ContainerDied","Data":"fd2ba366537ba0018a1dfdb4b6a2da2ef7cd69292a441710b9671a21520ddb69"}
Sep 17 17:54:17.553877 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.553852    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:54:17.737812 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.737731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-home\") pod \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") "
Sep 17 17:54:17.737812 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.737780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-dshm\") pod \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") "
Sep 17 17:54:17.738029 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.737819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tmp-dir\") pod \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") "
Sep 17 17:54:17.738029 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.737865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-model-cache\") pod \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") "
Sep 17 17:54:17.738029 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.737891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vzpt\" (UniqueName: \"kubernetes.io/projected/07c9d8a1-5319-43f0-80a2-4af08d2022f2-kube-api-access-2vzpt\") pod \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") "
Sep 17 17:54:17.738029 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.737921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tls-certs\") pod \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\" (UID: \"07c9d8a1-5319-43f0-80a2-4af08d2022f2\") "
Sep 17 17:54:17.738029 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.738008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-home" (OuterVolumeSpecName: "home") pod "07c9d8a1-5319-43f0-80a2-4af08d2022f2" (UID: "07c9d8a1-5319-43f0-80a2-4af08d2022f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:17.738331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.738119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "07c9d8a1-5319-43f0-80a2-4af08d2022f2" (UID: "07c9d8a1-5319-43f0-80a2-4af08d2022f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:17.738331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.738140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07c9d8a1-5319-43f0-80a2-4af08d2022f2" (UID: "07c9d8a1-5319-43f0-80a2-4af08d2022f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:17.738331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.738294    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:17.738331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.738307    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:17.738331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.738315    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:17.740174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.740116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c9d8a1-5319-43f0-80a2-4af08d2022f2-kube-api-access-2vzpt" (OuterVolumeSpecName: "kube-api-access-2vzpt") pod "07c9d8a1-5319-43f0-80a2-4af08d2022f2" (UID: "07c9d8a1-5319-43f0-80a2-4af08d2022f2"). InnerVolumeSpecName "kube-api-access-2vzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:17.740304 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.740207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07c9d8a1-5319-43f0-80a2-4af08d2022f2" (UID: "07c9d8a1-5319-43f0-80a2-4af08d2022f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:17.740304 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.740265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-dshm" (OuterVolumeSpecName: "dshm") pod "07c9d8a1-5319-43f0-80a2-4af08d2022f2" (UID: "07c9d8a1-5319-43f0-80a2-4af08d2022f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:17.839364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.839328    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c9d8a1-5319-43f0-80a2-4af08d2022f2-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:17.839364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.839361    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vzpt\" (UniqueName: \"kubernetes.io/projected/07c9d8a1-5319-43f0-80a2-4af08d2022f2-kube-api-access-2vzpt\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:17.839364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:17.839371    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c9d8a1-5319-43f0-80a2-4af08d2022f2-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:18.248566 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:18.248510    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:18.479192 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:18.479163    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"
Sep 17 17:54:18.479192 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:18.479180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7" event={"ID":"07c9d8a1-5319-43f0-80a2-4af08d2022f2","Type":"ContainerDied","Data":"1ac8ce96554136dfd186e848820202297fba21eea72bf1666a04d674f39be3a2"}
Sep 17 17:54:18.479398 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:18.479240    2580 scope.go:117] "RemoveContainer" containerID="fd2ba366537ba0018a1dfdb4b6a2da2ef7cd69292a441710b9671a21520ddb69"
Sep 17 17:54:18.497038 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:18.497015    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"]
Sep 17 17:54:18.501217 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:18.501196    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2pns7"]
Sep 17 17:54:20.046546 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:20.046511    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" path="/var/lib/kubelet/pods/07c9d8a1-5319-43f0-80a2-4af08d2022f2/volumes"
Sep 17 17:54:28.248146 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.248110    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:54:28.313314 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.313283    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 17:54:28.313409 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:54:28.313315    2580 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerName="main" containerID="cri-o://0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940"
Sep 17 17:54:28.313409 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.313357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerName="main" containerID="cri-o://0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940" gracePeriod=49
Sep 17 17:54:28.468499 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.468478    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52_cc7de744-5d30-4a2e-8744-55ead34930d3/main/0.log"
Sep 17 17:54:28.470797 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.470781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:54:28.533257 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.533187    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52_cc7de744-5d30-4a2e-8744-55ead34930d3/main/0.log"
Sep 17 17:54:28.533257 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.533229    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerID="0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940" exitCode=1
Sep 17 17:54:28.533415 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.533276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"
Sep 17 17:54:28.533415 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.533297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" event={"ID":"cc7de744-5d30-4a2e-8744-55ead34930d3","Type":"ContainerDied","Data":"0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940"}
Sep 17 17:54:28.533415 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.533347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52" event={"ID":"cc7de744-5d30-4a2e-8744-55ead34930d3","Type":"ContainerDied","Data":"276c940f8ca5d5849458ebceb3a731c0d25c3149404f85680ebe6eff72666867"}
Sep 17 17:54:28.533415 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.533376    2580 scope.go:117] "RemoveContainer" containerID="0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940"
Sep 17 17:54:28.536533 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.536509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7xkx\" (UniqueName: \"kubernetes.io/projected/cc7de744-5d30-4a2e-8744-55ead34930d3-kube-api-access-l7xkx\") pod \"cc7de744-5d30-4a2e-8744-55ead34930d3\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") "
Sep 17 17:54:28.536656 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.536593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7de744-5d30-4a2e-8744-55ead34930d3-tls-certs\") pod \"cc7de744-5d30-4a2e-8744-55ead34930d3\" (UID: \"cc7de744-5d30-4a2e-8744-55ead34930d3\") "
Sep 17 17:54:28.538582 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.538558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7de744-5d30-4a2e-8744-55ead34930d3-kube-api-access-l7xkx" (OuterVolumeSpecName: "kube-api-access-l7xkx") pod "cc7de744-5d30-4a2e-8744-55ead34930d3" (UID: "cc7de744-5d30-4a2e-8744-55ead34930d3"). InnerVolumeSpecName "kube-api-access-l7xkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:28.538709 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.538683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7de744-5d30-4a2e-8744-55ead34930d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc7de744-5d30-4a2e-8744-55ead34930d3" (UID: "cc7de744-5d30-4a2e-8744-55ead34930d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:28.542168 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.542150    2580 scope.go:117] "RemoveContainer" containerID="0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940"
Sep 17 17:54:28.542429 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:54:28.542410    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940\": container with ID starting with 0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940 not found: ID does not exist" containerID="0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940"
Sep 17 17:54:28.542497 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.542436    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940"} err="failed to get container status \"0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940\": rpc error: code = NotFound desc = could not find container \"0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940\": container with ID starting with 0547e0486861e0dc93cdcd465ad9bf95b6272ea38c8870e930d36ddbdeb36940 not found: ID does not exist"
Sep 17 17:54:28.637518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.637493    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7de744-5d30-4a2e-8744-55ead34930d3-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:28.637518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.637516    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7xkx\" (UniqueName: \"kubernetes.io/projected/cc7de744-5d30-4a2e-8744-55ead34930d3-kube-api-access-l7xkx\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:28.856004 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.855970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"]
Sep 17 17:54:28.859906 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:28.859881    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5hhs52"]
Sep 17 17:54:30.045956 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:30.045922    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" path="/var/lib/kubelet/pods/cc7de744-5d30-4a2e-8744-55ead34930d3/volumes"
Sep 17 17:54:38.234907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.234873    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"]
Sep 17 17:54:38.237336 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.237297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:38.244352 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:54:38.244321    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:54:38.244352 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.244352 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.244352 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="28151c11c6b3fa09f92717eeb70866b71afbc6c725c4fd0c0e7e1b478c0fd2de" cmd=["/bin/sleep","15"]
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.244372    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" message="" err=<
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:54:38.244389    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" podUID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" containerName="main" containerID="cri-o://28151c11c6b3fa09f92717eeb70866b71afbc6c725c4fd0c0e7e1b478c0fd2de"
Sep 17 17:54:38.244606 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.244432    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" podUID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" containerName="main" containerID="cri-o://28151c11c6b3fa09f92717eeb70866b71afbc6c725c4fd0c0e7e1b478c0fd2de" gracePeriod=60
Sep 17 17:54:38.303259 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.303226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:38.419334 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.419303    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" podUID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 17:54:38.594707 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.594611    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" containerID="28151c11c6b3fa09f92717eeb70866b71afbc6c725c4fd0c0e7e1b478c0fd2de" exitCode=0
Sep 17 17:54:38.594707 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:38.594667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" event={"ID":"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5","Type":"ContainerDied","Data":"28151c11c6b3fa09f92717eeb70866b71afbc6c725c4fd0c0e7e1b478c0fd2de"}
Sep 17 17:54:39.003175 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.003149    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:54:39.035722 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.035689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:39.125516 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-dshm\") pod \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") "
Sep 17 17:54:39.125516 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-home\") pod \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") "
Sep 17 17:54:39.125754 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhwlj\" (UniqueName: \"kubernetes.io/projected/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-kube-api-access-hhwlj\") pod \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") "
Sep 17 17:54:39.125754 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125553    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tmp-dir\") pod \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") "
Sep 17 17:54:39.125754 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125573    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-model-cache\") pod \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") "
Sep 17 17:54:39.125754 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125588    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tls-certs\") pod \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\" (UID: \"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5\") "
Sep 17 17:54:39.125945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125803    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-home" (OuterVolumeSpecName: "home") pod "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" (UID: "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.125945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" (UID: "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.125945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.125880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" (UID: "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.127859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.127830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" (UID: "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:39.127859 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.127838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-kube-api-access-hhwlj" (OuterVolumeSpecName: "kube-api-access-hhwlj") pod "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" (UID: "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5"). InnerVolumeSpecName "kube-api-access-hhwlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:39.128011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.127906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-dshm" (OuterVolumeSpecName: "dshm") pod "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" (UID: "aa3b9fb8-3517-49f2-84bb-8eb9923b31a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.227148 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.227117    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwlj\" (UniqueName: \"kubernetes.io/projected/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-kube-api-access-hhwlj\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.227310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.227161    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.227310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.227171    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.227310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.227180    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.227310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.227188    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.227310 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.227195    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.602864 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.602840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"
Sep 17 17:54:39.603285 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.602835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl" event={"ID":"aa3b9fb8-3517-49f2-84bb-8eb9923b31a5","Type":"ContainerDied","Data":"69945953923c0fe7fdcc26f0942d3140dbf299e3e9dde45136567f2a6c3e910e"}
Sep 17 17:54:39.603285 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.602973    2580 scope.go:117] "RemoveContainer" containerID="28151c11c6b3fa09f92717eeb70866b71afbc6c725c4fd0c0e7e1b478c0fd2de"
Sep 17 17:54:39.625257 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.625232    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"]
Sep 17 17:54:39.628670 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:39.628650    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7bghl"]
Sep 17 17:54:40.046436 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:54:40.046402    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" path="/var/lib/kubelet/pods/aa3b9fb8-3517-49f2-84bb-8eb9923b31a5/volumes"
Sep 17 17:55:06.087279 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:06.087241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:06.183125 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:06.183070    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:09.176219 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176191    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"]
Sep 17 17:55:09.176576 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176531    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerName="main"
Sep 17 17:55:09.176576 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176542    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerName="main"
Sep 17 17:55:09.176576 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176556    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" containerName="main"
Sep 17 17:55:09.176576 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176562    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" containerName="main"
Sep 17 17:55:09.176576 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176576    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" containerName="main"
Sep 17 17:55:09.176736 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176582    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" containerName="main"
Sep 17 17:55:09.176736 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176642    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c9d8a1-5319-43f0-80a2-4af08d2022f2" containerName="main"
Sep 17 17:55:09.176736 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176648    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7de744-5d30-4a2e-8744-55ead34930d3" containerName="main"
Sep 17 17:55:09.176736 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.176654    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3b9fb8-3517-49f2-84bb-8eb9923b31a5" containerName="main"
Sep 17 17:55:09.179754 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.179735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.187889 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.187867    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 17:55:09.188209 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.188194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 17:55:09.189533 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.189520    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-mn96v\""
Sep 17 17:55:09.192788 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.192773    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 17:55:09.211838 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.211817    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"]
Sep 17 17:55:09.268189 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.268156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5f3a-871b-4b33-acb4-ff174104054a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.268331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.268209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.268331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.268276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.268331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.268310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.268500 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.268337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.268500 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.268355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn6d\" (UniqueName: \"kubernetes.io/projected/1d9b5f3a-871b-4b33-acb4-ff174104054a-kube-api-access-9jn6d\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.272111 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.272065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:09.369348 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5f3a-871b-4b33-acb4-ff174104054a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369599 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369599 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369599 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn6d\" (UniqueName: \"kubernetes.io/projected/1d9b5f3a-871b-4b33-acb4-ff174104054a-kube-api-access-9jn6d\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369721 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369824 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.369960 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.369853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.371680 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.371663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.371996 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.371979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5f3a-871b-4b33-acb4-ff174104054a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.381264 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.381239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn6d\" (UniqueName: \"kubernetes.io/projected/1d9b5f3a-871b-4b33-acb4-ff174104054a-kube-api-access-9jn6d\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ggkw8\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.520164 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.520076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:09.546661 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.546632    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:09.796130 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:09.796101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"]
Sep 17 17:55:09.797865 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:55:09.797841    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9b5f3a_871b_4b33_acb4_ff174104054a.slice/crio-fbccd02b6fe5e51b6e97842e5fc03665ae339aa8bee938ce170c617def6c47ef WatchSource:0}: Error finding container fbccd02b6fe5e51b6e97842e5fc03665ae339aa8bee938ce170c617def6c47ef: Status 404 returned error can't find the container with id fbccd02b6fe5e51b6e97842e5fc03665ae339aa8bee938ce170c617def6c47ef
Sep 17 17:55:10.781104 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:10.781044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" event={"ID":"1d9b5f3a-871b-4b33-acb4-ff174104054a","Type":"ContainerStarted","Data":"48da17687bc1fb6f3619ed44db2cb0c87b0e060b72dfa356a05b847eaffd0303"}
Sep 17 17:55:10.781526 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:10.781123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" event={"ID":"1d9b5f3a-871b-4b33-acb4-ff174104054a","Type":"ContainerStarted","Data":"fbccd02b6fe5e51b6e97842e5fc03665ae339aa8bee938ce170c617def6c47ef"}
Sep 17 17:55:10.811478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:10.811429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" podStartSLOduration=1.811412347 podStartE2EDuration="1.811412347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:10.810237559 +0000 UTC m=+887.149631151" watchObservedRunningTime="2026-09-17 17:55:10.811412347 +0000 UTC m=+887.150805938"
Sep 17 17:55:11.520929 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:11.520899    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:13.505153 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:13.505109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:19.520361 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:19.520281    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:19.537273 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:19.537252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:19.583027 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:19.582993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:19.732191 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:19.732158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:19.847106 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:19.847014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:55:25.061293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:25.061266    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:55:25.065317 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:25.065296    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 17:55:28.309293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:28.309258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:28.411250 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:28.411208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:35.287157 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:35.287117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:41.307734 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:41.307697    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:41.411473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:41.411434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:43.229430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:55:43.229396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:03.895155 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.895120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"]
Sep 17 17:56:03.898730 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.898708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:03.902548 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.902527    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 17:56:03.902680 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.902608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 17:56:03.902680 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.902629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 17:56:03.902680 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.902636    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-5vnpl\""
Sep 17 17:56:03.908716 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.908691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"]
Sep 17 17:56:03.936622 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:03.936594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:04.032790 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.032760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.032932 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.032841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32306397-df80-4eb3-aa00-0baf7ba674f8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.032932 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.032876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.032932 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.032894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4cn\" (UniqueName: \"kubernetes.io/projected/32306397-df80-4eb3-aa00-0baf7ba674f8-kube-api-access-vb4cn\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.032932 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.032917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.033075 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.032938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134045 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32306397-df80-4eb3-aa00-0baf7ba674f8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134222 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134222 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4cn\" (UniqueName: \"kubernetes.io/projected/32306397-df80-4eb3-aa00-0baf7ba674f8-kube-api-access-vb4cn\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134222 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134222 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134450 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134516 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134595 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.134595 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.134566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.136363 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.136342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.136687 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.136670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32306397-df80-4eb3-aa00-0baf7ba674f8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.142873 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.142849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4cn\" (UniqueName: \"kubernetes.io/projected/32306397-df80-4eb3-aa00-0baf7ba674f8-kube-api-access-vb4cn\") pod \"fc-concurrency-test-kserve-b5d44467c-wmhx6\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.152217 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.152139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:04.209815 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.209786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:04.489847 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:04.489823    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"]
Sep 17 17:56:04.491642 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:56:04.491609    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32306397_df80_4eb3_aa00_0baf7ba674f8.slice/crio-6071e454cb7e243bc621d90dda57e60bb84f02bfda5f0074e041bbd01257d4d2 WatchSource:0}: Error finding container 6071e454cb7e243bc621d90dda57e60bb84f02bfda5f0074e041bbd01257d4d2: Status 404 returned error can't find the container with id 6071e454cb7e243bc621d90dda57e60bb84f02bfda5f0074e041bbd01257d4d2
Sep 17 17:56:05.083548 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:05.083513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" event={"ID":"32306397-df80-4eb3-aa00-0baf7ba674f8","Type":"ContainerStarted","Data":"f7365e2839d39e5e20e2522ee7ec733f230b4d5f5f5ab9393399b92cd090d9e0"}
Sep 17 17:56:05.083928 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:05.083570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" event={"ID":"32306397-df80-4eb3-aa00-0baf7ba674f8","Type":"ContainerStarted","Data":"6071e454cb7e243bc621d90dda57e60bb84f02bfda5f0074e041bbd01257d4d2"}
Sep 17 17:56:05.105305 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:05.105265    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" podStartSLOduration=2.105250674 podStartE2EDuration="2.105250674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:05.103642398 +0000 UTC m=+941.443036001" watchObservedRunningTime="2026-09-17 17:56:05.105250674 +0000 UTC m=+941.444644265"
Sep 17 17:56:05.210138 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:05.210097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:06.732578 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:06.732539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:06.738897 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:06.738867    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"]
Sep 17 17:56:06.749746 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:56:06.749712    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:56:06.749746 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.749746 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.749746 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="48da17687bc1fb6f3619ed44db2cb0c87b0e060b72dfa356a05b847eaffd0303" cmd=["/bin/sleep","15"]
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:06.749757    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" message="" err=<
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:56:06.749770    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" podUID="1d9b5f3a-871b-4b33-acb4-ff174104054a" containerName="main" containerID="cri-o://48da17687bc1fb6f3619ed44db2cb0c87b0e060b72dfa356a05b847eaffd0303"
Sep 17 17:56:06.749980 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:06.749807    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" podUID="1d9b5f3a-871b-4b33-acb4-ff174104054a" containerName="main" containerID="cri-o://48da17687bc1fb6f3619ed44db2cb0c87b0e060b72dfa356a05b847eaffd0303" gracePeriod=60
Sep 17 17:56:06.833293 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:06.833259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:06.836843 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:06.836812    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" podUID="1d9b5f3a-871b-4b33-acb4-ff174104054a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 17:56:07.096739 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.096654    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d9b5f3a-871b-4b33-acb4-ff174104054a" containerID="48da17687bc1fb6f3619ed44db2cb0c87b0e060b72dfa356a05b847eaffd0303" exitCode=0
Sep 17 17:56:07.096739 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.096722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" event={"ID":"1d9b5f3a-871b-4b33-acb4-ff174104054a","Type":"ContainerDied","Data":"48da17687bc1fb6f3619ed44db2cb0c87b0e060b72dfa356a05b847eaffd0303"}
Sep 17 17:56:07.503446 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.503420    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:56:07.664640 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.664606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-dshm\") pod \"1d9b5f3a-871b-4b33-acb4-ff174104054a\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") "
Sep 17 17:56:07.664855 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.664654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jn6d\" (UniqueName: \"kubernetes.io/projected/1d9b5f3a-871b-4b33-acb4-ff174104054a-kube-api-access-9jn6d\") pod \"1d9b5f3a-871b-4b33-acb4-ff174104054a\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") "
Sep 17 17:56:07.664855 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.664688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-tmp-dir\") pod \"1d9b5f3a-871b-4b33-acb4-ff174104054a\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") "
Sep 17 17:56:07.664855 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.664787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5f3a-871b-4b33-acb4-ff174104054a-tls-certs\") pod \"1d9b5f3a-871b-4b33-acb4-ff174104054a\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") "
Sep 17 17:56:07.664855 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.664819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-model-cache\") pod \"1d9b5f3a-871b-4b33-acb4-ff174104054a\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") "
Sep 17 17:56:07.664855 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.664843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-home\") pod \"1d9b5f3a-871b-4b33-acb4-ff174104054a\" (UID: \"1d9b5f3a-871b-4b33-acb4-ff174104054a\") "
Sep 17 17:56:07.665140 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.665057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d9b5f3a-871b-4b33-acb4-ff174104054a" (UID: "1d9b5f3a-871b-4b33-acb4-ff174104054a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.665140 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.665069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d9b5f3a-871b-4b33-acb4-ff174104054a" (UID: "1d9b5f3a-871b-4b33-acb4-ff174104054a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.665356 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.665331    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-home" (OuterVolumeSpecName: "home") pod "1d9b5f3a-871b-4b33-acb4-ff174104054a" (UID: "1d9b5f3a-871b-4b33-acb4-ff174104054a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.666962 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.666935    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-dshm" (OuterVolumeSpecName: "dshm") pod "1d9b5f3a-871b-4b33-acb4-ff174104054a" (UID: "1d9b5f3a-871b-4b33-acb4-ff174104054a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.667214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.667191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9b5f3a-871b-4b33-acb4-ff174104054a-kube-api-access-9jn6d" (OuterVolumeSpecName: "kube-api-access-9jn6d") pod "1d9b5f3a-871b-4b33-acb4-ff174104054a" (UID: "1d9b5f3a-871b-4b33-acb4-ff174104054a"). InnerVolumeSpecName "kube-api-access-9jn6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:07.667214 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.667198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b5f3a-871b-4b33-acb4-ff174104054a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d9b5f3a-871b-4b33-acb4-ff174104054a" (UID: "1d9b5f3a-871b-4b33-acb4-ff174104054a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:07.766165 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.766125    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5f3a-871b-4b33-acb4-ff174104054a-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.766165 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.766162    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.766165 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.766171    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.766605 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.766178    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.766605 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.766187    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jn6d\" (UniqueName: \"kubernetes.io/projected/1d9b5f3a-871b-4b33-acb4-ff174104054a-kube-api-access-9jn6d\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.766605 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:07.766197    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5f3a-871b-4b33-acb4-ff174104054a-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:08.103556 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:08.103520    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"
Sep 17 17:56:08.103556 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:08.103544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8" event={"ID":"1d9b5f3a-871b-4b33-acb4-ff174104054a","Type":"ContainerDied","Data":"fbccd02b6fe5e51b6e97842e5fc03665ae339aa8bee938ce170c617def6c47ef"}
Sep 17 17:56:08.103721 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:08.103599    2580 scope.go:117] "RemoveContainer" containerID="48da17687bc1fb6f3619ed44db2cb0c87b0e060b72dfa356a05b847eaffd0303"
Sep 17 17:56:08.120547 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:08.120524    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"]
Sep 17 17:56:08.124479 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:08.124455    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ggkw8"]
Sep 17 17:56:10.045514 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:10.045480    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9b5f3a-871b-4b33-acb4-ff174104054a" path="/var/lib/kubelet/pods/1d9b5f3a-871b-4b33-acb4-ff174104054a/volumes"
Sep 17 17:56:14.210741 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:14.210710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:14.223246 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:14.223222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:15.153812 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:15.153787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:18.396569 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:18.396539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:43.554752 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:43.554702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:43.581518 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:43.581488    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"]
Sep 17 17:56:43.590423 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:56:43.590386    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:56:43.590423 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:43.590423 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:43.590423 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="f7365e2839d39e5e20e2522ee7ec733f230b4d5f5f5ab9393399b92cd090d9e0" cmd=["/bin/sleep","15"]
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:43.590433    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" message="" err=<
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:56:43.590444    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" podUID="32306397-df80-4eb3-aa00-0baf7ba674f8" containerName="main" containerID="cri-o://f7365e2839d39e5e20e2522ee7ec733f230b4d5f5f5ab9393399b92cd090d9e0"
Sep 17 17:56:43.590660 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:43.590488    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" podUID="32306397-df80-4eb3-aa00-0baf7ba674f8" containerName="main" containerID="cri-o://f7365e2839d39e5e20e2522ee7ec733f230b4d5f5f5ab9393399b92cd090d9e0" gracePeriod=60
Sep 17 17:56:43.648767 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:43.648732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:44.143383 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.143343    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" podUID="32306397-df80-4eb3-aa00-0baf7ba674f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 17:56:44.299135 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.299043    2580 generic.go:356] "Generic (PLEG): container finished" podID="32306397-df80-4eb3-aa00-0baf7ba674f8" containerID="f7365e2839d39e5e20e2522ee7ec733f230b4d5f5f5ab9393399b92cd090d9e0" exitCode=0
Sep 17 17:56:44.299135 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.299121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" event={"ID":"32306397-df80-4eb3-aa00-0baf7ba674f8","Type":"ContainerDied","Data":"f7365e2839d39e5e20e2522ee7ec733f230b4d5f5f5ab9393399b92cd090d9e0"}
Sep 17 17:56:44.347112 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.347075    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:44.466873 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.466790    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-home\") pod \"32306397-df80-4eb3-aa00-0baf7ba674f8\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") "
Sep 17 17:56:44.466873 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.466831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-tmp-dir\") pod \"32306397-df80-4eb3-aa00-0baf7ba674f8\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") "
Sep 17 17:56:44.466873 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.466866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-dshm\") pod \"32306397-df80-4eb3-aa00-0baf7ba674f8\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") "
Sep 17 17:56:44.467174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.466883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-model-cache\") pod \"32306397-df80-4eb3-aa00-0baf7ba674f8\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") "
Sep 17 17:56:44.467174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.466910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32306397-df80-4eb3-aa00-0baf7ba674f8-tls-certs\") pod \"32306397-df80-4eb3-aa00-0baf7ba674f8\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") "
Sep 17 17:56:44.467174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.466993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb4cn\" (UniqueName: \"kubernetes.io/projected/32306397-df80-4eb3-aa00-0baf7ba674f8-kube-api-access-vb4cn\") pod \"32306397-df80-4eb3-aa00-0baf7ba674f8\" (UID: \"32306397-df80-4eb3-aa00-0baf7ba674f8\") "
Sep 17 17:56:44.467174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.467125    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-home" (OuterVolumeSpecName: "home") pod "32306397-df80-4eb3-aa00-0baf7ba674f8" (UID: "32306397-df80-4eb3-aa00-0baf7ba674f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:44.467389 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.467215    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "32306397-df80-4eb3-aa00-0baf7ba674f8" (UID: "32306397-df80-4eb3-aa00-0baf7ba674f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:44.467389 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.467232    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32306397-df80-4eb3-aa00-0baf7ba674f8" (UID: "32306397-df80-4eb3-aa00-0baf7ba674f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:44.467389 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.467255    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:44.469131 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.469101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-dshm" (OuterVolumeSpecName: "dshm") pod "32306397-df80-4eb3-aa00-0baf7ba674f8" (UID: "32306397-df80-4eb3-aa00-0baf7ba674f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:44.469231 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.469178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32306397-df80-4eb3-aa00-0baf7ba674f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32306397-df80-4eb3-aa00-0baf7ba674f8" (UID: "32306397-df80-4eb3-aa00-0baf7ba674f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:44.469231 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.469212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32306397-df80-4eb3-aa00-0baf7ba674f8-kube-api-access-vb4cn" (OuterVolumeSpecName: "kube-api-access-vb4cn") pod "32306397-df80-4eb3-aa00-0baf7ba674f8" (UID: "32306397-df80-4eb3-aa00-0baf7ba674f8"). InnerVolumeSpecName "kube-api-access-vb4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:44.568397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.568370    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb4cn\" (UniqueName: \"kubernetes.io/projected/32306397-df80-4eb3-aa00-0baf7ba674f8-kube-api-access-vb4cn\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:44.568397 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.568395    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:44.568737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.568405    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:44.568737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.568413    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32306397-df80-4eb3-aa00-0baf7ba674f8-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:44.568737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:44.568422    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32306397-df80-4eb3-aa00-0baf7ba674f8-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:56:45.305830 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:45.305805    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"
Sep 17 17:56:45.305995 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:45.305802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6" event={"ID":"32306397-df80-4eb3-aa00-0baf7ba674f8","Type":"ContainerDied","Data":"6071e454cb7e243bc621d90dda57e60bb84f02bfda5f0074e041bbd01257d4d2"}
Sep 17 17:56:45.305995 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:45.305942    2580 scope.go:117] "RemoveContainer" containerID="f7365e2839d39e5e20e2522ee7ec733f230b4d5f5f5ab9393399b92cd090d9e0"
Sep 17 17:56:45.329299 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:45.329274    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"]
Sep 17 17:56:45.332931 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:45.332908    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wmhx6"]
Sep 17 17:56:46.045592 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:46.045563    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32306397-df80-4eb3-aa00-0baf7ba674f8" path="/var/lib/kubelet/pods/32306397-df80-4eb3-aa00-0baf7ba674f8/volumes"
Sep 17 17:56:55.645251 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:55.645211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:55.739799 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:56:55.739762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:01.039707 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.039676    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"]
Sep 17 17:57:01.040188 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.040176    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b5f3a-871b-4b33-acb4-ff174104054a" containerName="main"
Sep 17 17:57:01.040267 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.040193    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b5f3a-871b-4b33-acb4-ff174104054a" containerName="main"
Sep 17 17:57:01.040267 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.040201    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32306397-df80-4eb3-aa00-0baf7ba674f8" containerName="main"
Sep 17 17:57:01.040267 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.040207    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32306397-df80-4eb3-aa00-0baf7ba674f8" containerName="main"
Sep 17 17:57:01.040405 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.040295    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32306397-df80-4eb3-aa00-0baf7ba674f8" containerName="main"
Sep 17 17:57:01.040405 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.040308    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9b5f3a-871b-4b33-acb4-ff174104054a" containerName="main"
Sep 17 17:57:01.043611 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.043586    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.047541 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.047502    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-nsgbc\""
Sep 17 17:57:01.047689 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.047504    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 17:57:01.047689 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.047548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 17:57:01.047822 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.047779    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-kfx4j\""
Sep 17 17:57:01.047822 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.047806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 17:57:01.058688 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.058575    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"]
Sep 17 17:57:01.063168 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.063145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"]
Sep 17 17:57:01.068719 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.068697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.079047 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.079022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"]
Sep 17 17:57:01.099053 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.099027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c951a7ff-e40f-4b17-a313-2d961e94eca6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.099169 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.099074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.099169 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.099115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.099169 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.099139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrks2\" (UniqueName: \"kubernetes.io/projected/c951a7ff-e40f-4b17-a313-2d961e94eca6-kube-api-access-zrks2\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.099349 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.099197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.099349 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.099245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.113377 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.113348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:01.199775 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.199908 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.199908 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.199908 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrks2\" (UniqueName: \"kubernetes.io/projected/c951a7ff-e40f-4b17-a313-2d961e94eca6-kube-api-access-zrks2\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.199908 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.199908 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.200194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.200194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.199993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.200194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.200017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.200194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.200047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpmd\" (UniqueName: \"kubernetes.io/projected/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-kube-api-access-rxpmd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.200194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.200107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.200194 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.200155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.200480 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.200215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c951a7ff-e40f-4b17-a313-2d961e94eca6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.200480 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.200342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.200480 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.200341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.202127 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.202111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.202449 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.202433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c951a7ff-e40f-4b17-a313-2d961e94eca6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.207710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.207694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrks2\" (UniqueName: \"kubernetes.io/projected/c951a7ff-e40f-4b17-a313-2d961e94eca6-kube-api-access-zrks2\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.300865 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.300793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.300865 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.300830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpmd\" (UniqueName: \"kubernetes.io/projected/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-kube-api-access-rxpmd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.300865 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.300856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.301156 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.300892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.301156 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.300914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.301156 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.300933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.301577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.301205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.301577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.301302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.301577 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.301340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.303136 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.303066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.303384 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.303366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.312115 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.312063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpmd\" (UniqueName: \"kubernetes.io/projected/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-kube-api-access-rxpmd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.324945 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.324885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:01.359530 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.359511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:01.380617 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.380597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:01.637834 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.637809    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"]
Sep 17 17:57:01.639474 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:57:01.639444    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc951a7ff_e40f_4b17_a313_2d961e94eca6.slice/crio-8b09369d11b0bab795e25450182959d519c1d541ac45aea21efaa03bc61e730a WatchSource:0}: Error finding container 8b09369d11b0bab795e25450182959d519c1d541ac45aea21efaa03bc61e730a: Status 404 returned error can't find the container with id 8b09369d11b0bab795e25450182959d519c1d541ac45aea21efaa03bc61e730a
Sep 17 17:57:01.641997 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.641974    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:57:01.665346 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:01.665320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"]
Sep 17 17:57:01.666336 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:57:01.666313    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda866688_d33c_4c9c_b6b9_3d3d4ba7c0ad.slice/crio-58f1c6a823d711fba3fd9c34ecf1b63198cfae08b1224738eb9350aeedf509fa WatchSource:0}: Error finding container 58f1c6a823d711fba3fd9c34ecf1b63198cfae08b1224738eb9350aeedf509fa: Status 404 returned error can't find the container with id 58f1c6a823d711fba3fd9c34ecf1b63198cfae08b1224738eb9350aeedf509fa
Sep 17 17:57:02.402119 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:02.402052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" event={"ID":"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad","Type":"ContainerStarted","Data":"354d6d6ecd0c6a18a1a44c0f675fb86dfa84a0c5ea030771c1d6aee71f2f11f9"}
Sep 17 17:57:02.402592 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:02.402128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" event={"ID":"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad","Type":"ContainerStarted","Data":"58f1c6a823d711fba3fd9c34ecf1b63198cfae08b1224738eb9350aeedf509fa"}
Sep 17 17:57:02.403524 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:02.403491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" event={"ID":"c951a7ff-e40f-4b17-a313-2d961e94eca6","Type":"ContainerStarted","Data":"8b09369d11b0bab795e25450182959d519c1d541ac45aea21efaa03bc61e730a"}
Sep 17 17:57:02.422279 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:02.422232    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" podStartSLOduration=1.422213023 podStartE2EDuration="1.422213023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:02.420128888 +0000 UTC m=+998.759522481" watchObservedRunningTime="2026-09-17 17:57:02.422213023 +0000 UTC m=+998.761606615"
Sep 17 17:57:03.381347 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:03.381269    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:03.410769 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:03.410731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" event={"ID":"c951a7ff-e40f-4b17-a313-2d961e94eca6","Type":"ContainerStarted","Data":"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"}
Sep 17 17:57:03.411196 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:03.410912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:03.651640 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:03.651604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:04.418240 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:04.418206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" event={"ID":"c951a7ff-e40f-4b17-a313-2d961e94eca6","Type":"ContainerStarted","Data":"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"}
Sep 17 17:57:04.440712 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:04.440665    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" podStartSLOduration=2.00797771 podStartE2EDuration="3.440650367s" totalImagesPullingTime="1.432672657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:04.439763765 +0000 UTC m=+1000.779157356" watchObservedRunningTime="2026-09-17 17:57:04.440650367 +0000 UTC m=+1000.780043957"
Sep 17 17:57:05.359766 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:05.359733    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:10.074498 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:10.074466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:10.173399 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:10.173372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:11.359733 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:11.359697    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:11.371737 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:11.371709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:11.381569 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:11.381544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:11.393707 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:11.393684    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:11.469690 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:11.469665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:11.470503 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:11.470487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:57:17.240387 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:17.240351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:22.478067 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:22.478032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:57:33.438962 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:33.438930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:33.538174 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:33.538142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:33.642440 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:33.642402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:33.764140 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:33.764103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:33.859439 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:33.859407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:37.363124 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:37.363071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:40.162907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:40.162871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:42.688270 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:42.688227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:45.283978 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:45.283940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:45.413897 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:45.413851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:45.689813 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:45.689779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:45.906036 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:45.906002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:48.581718 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:48.581680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:49.108143 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:49.108105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:49.207881 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:49.207847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:49.308513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:49.308473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:50.198532 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:50.198499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:54.335309 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:54.335275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:55.976374 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:55.976320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:57.172777 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:57.170399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:57.611450 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:57.611363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:57.700364 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:57.700338    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:59.137724 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:57:59.137677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:02.323850 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:02.323818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:02.692041 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:02.692004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:03.579835 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:03.579803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:03.672448 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:03.672415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:11.889746 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:11.889709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:12.274550 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:12.274508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:24.169190 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:24.169146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:24.908230 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:24.908192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:28.397383 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:28.397341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:30.333894 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:30.333830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:31.008378 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:31.008344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:31.106812 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:31.106774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:42.442810 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:42.442774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.712450 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.712413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.743710 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.743679    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"]
Sep 17 17:58:46.746592 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.746559    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"]
Sep 17 17:58:46.753927 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:58:46.753899    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:58:46.753927 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:46.753927 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:46.753927 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="354d6d6ecd0c6a18a1a44c0f675fb86dfa84a0c5ea030771c1d6aee71f2f11f9" cmd=["/bin/sleep","15"]
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.753944    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" message="" err=<
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:58:46.753955    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" podUID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" containerName="main" containerID="cri-o://354d6d6ecd0c6a18a1a44c0f675fb86dfa84a0c5ea030771c1d6aee71f2f11f9"
Sep 17 17:58:46.754176 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.753999    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" podUID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" containerName="main" containerID="cri-o://354d6d6ecd0c6a18a1a44c0f675fb86dfa84a0c5ea030771c1d6aee71f2f11f9" gracePeriod=60
Sep 17 17:58:46.782621 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.782594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.975896 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.975841    2580 generic.go:356] "Generic (PLEG): container finished" podID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" containerID="354d6d6ecd0c6a18a1a44c0f675fb86dfa84a0c5ea030771c1d6aee71f2f11f9" exitCode=0
Sep 17 17:58:46.975988 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.975910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" event={"ID":"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad","Type":"ContainerDied","Data":"354d6d6ecd0c6a18a1a44c0f675fb86dfa84a0c5ea030771c1d6aee71f2f11f9"}
Sep 17 17:58:46.983724 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:58:46.983699    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:58:46.983724 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:46.983724 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:46.983724 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b" cmd=["/bin/sleep","15"]
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.983739    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" message="" err=<
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:58:46.983751    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="main" containerID="cri-o://b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"
Sep 17 17:58:46.983909 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:46.983781    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="main" containerID="cri-o://b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b" gracePeriod=60
Sep 17 17:58:47.459630 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.459593    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
Sep 17 17:58:47.496302 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.496275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:58:47.598135 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.598057    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="llm-d-routing-sidecar" containerID="cri-o://e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef" gracePeriod=60
Sep 17 17:58:47.674478 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tls-certs\") pod \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") "
Sep 17 17:58:47.674606 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-dshm\") pod \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") "
Sep 17 17:58:47.674606 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-home\") pod \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") "
Sep 17 17:58:47.674606 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-model-cache\") pod \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") "
Sep 17 17:58:47.674606 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tmp-dir\") pod \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") "
Sep 17 17:58:47.674905 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxpmd\" (UniqueName: \"kubernetes.io/projected/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-kube-api-access-rxpmd\") pod \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\" (UID: \"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad\") "
Sep 17 17:58:47.674905 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" (UID: "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.675026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.674948    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" (UID: "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.675026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.675001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-home" (OuterVolumeSpecName: "home") pod "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" (UID: "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.675026 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.675023    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:47.675226 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.675040    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:47.676720 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.676694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" (UID: "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:47.676720 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.676697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-dshm" (OuterVolumeSpecName: "dshm") pod "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" (UID: "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.676893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.676877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-kube-api-access-rxpmd" (OuterVolumeSpecName: "kube-api-access-rxpmd") pod "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" (UID: "da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad"). InnerVolumeSpecName "kube-api-access-rxpmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:47.776283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.776257    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:47.776283 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.776279    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:47.776740 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.776289    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpmd\" (UniqueName: \"kubernetes.io/projected/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-kube-api-access-rxpmd\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:47.776740 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.776299    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:47.833905 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.833879    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:58:47.978268 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978207    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-model-cache\") pod \"c951a7ff-e40f-4b17-a313-2d961e94eca6\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") "
Sep 17 17:58:47.978268 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-home\") pod \"c951a7ff-e40f-4b17-a313-2d961e94eca6\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") "
Sep 17 17:58:47.978473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978283    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c951a7ff-e40f-4b17-a313-2d961e94eca6-tls-certs\") pod \"c951a7ff-e40f-4b17-a313-2d961e94eca6\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") "
Sep 17 17:58:47.978473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978312    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrks2\" (UniqueName: \"kubernetes.io/projected/c951a7ff-e40f-4b17-a313-2d961e94eca6-kube-api-access-zrks2\") pod \"c951a7ff-e40f-4b17-a313-2d961e94eca6\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") "
Sep 17 17:58:47.978473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-tmp-dir\") pod \"c951a7ff-e40f-4b17-a313-2d961e94eca6\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") "
Sep 17 17:58:47.978473 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-dshm\") pod \"c951a7ff-e40f-4b17-a313-2d961e94eca6\" (UID: \"c951a7ff-e40f-4b17-a313-2d961e94eca6\") "
Sep 17 17:58:47.978704 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c951a7ff-e40f-4b17-a313-2d961e94eca6" (UID: "c951a7ff-e40f-4b17-a313-2d961e94eca6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.978704 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-home" (OuterVolumeSpecName: "home") pod "c951a7ff-e40f-4b17-a313-2d961e94eca6" (UID: "c951a7ff-e40f-4b17-a313-2d961e94eca6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.978704 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.978663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c951a7ff-e40f-4b17-a313-2d961e94eca6" (UID: "c951a7ff-e40f-4b17-a313-2d961e94eca6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.980472 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.980441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c951a7ff-e40f-4b17-a313-2d961e94eca6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c951a7ff-e40f-4b17-a313-2d961e94eca6" (UID: "c951a7ff-e40f-4b17-a313-2d961e94eca6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:47.980472 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.980457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c951a7ff-e40f-4b17-a313-2d961e94eca6-kube-api-access-zrks2" (OuterVolumeSpecName: "kube-api-access-zrks2") pod "c951a7ff-e40f-4b17-a313-2d961e94eca6" (UID: "c951a7ff-e40f-4b17-a313-2d961e94eca6"). InnerVolumeSpecName "kube-api-access-zrks2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:47.980740 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.980712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-dshm" (OuterVolumeSpecName: "dshm") pod "c951a7ff-e40f-4b17-a313-2d961e94eca6" (UID: "c951a7ff-e40f-4b17-a313-2d961e94eca6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:47.983934 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.983915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"
Sep 17 17:58:47.984028 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.983929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" event={"ID":"da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad","Type":"ContainerDied","Data":"58f1c6a823d711fba3fd9c34ecf1b63198cfae08b1224738eb9350aeedf509fa"}
Sep 17 17:58:47.984028 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.984020    2580 scope.go:117] "RemoveContainer" containerID="354d6d6ecd0c6a18a1a44c0f675fb86dfa84a0c5ea030771c1d6aee71f2f11f9"
Sep 17 17:58:47.985893 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.985869    2580 generic.go:356] "Generic (PLEG): container finished" podID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerID="b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b" exitCode=0
Sep 17 17:58:47.986007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.985895    2580 generic.go:356] "Generic (PLEG): container finished" podID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerID="e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef" exitCode=0
Sep 17 17:58:47.986007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.985931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"
Sep 17 17:58:47.986007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.985946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" event={"ID":"c951a7ff-e40f-4b17-a313-2d961e94eca6","Type":"ContainerDied","Data":"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"}
Sep 17 17:58:47.986007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.985992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" event={"ID":"c951a7ff-e40f-4b17-a313-2d961e94eca6","Type":"ContainerDied","Data":"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"}
Sep 17 17:58:47.986260 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.986013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz" event={"ID":"c951a7ff-e40f-4b17-a313-2d961e94eca6","Type":"ContainerDied","Data":"8b09369d11b0bab795e25450182959d519c1d541ac45aea21efaa03bc61e730a"}
Sep 17 17:58:47.999954 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:47.999936    2580 scope.go:117] "RemoveContainer" containerID="b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"
Sep 17 17:58:48.007957 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.007940    2580 scope.go:117] "RemoveContainer" containerID="e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"
Sep 17 17:58:48.011834 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.011812    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"]
Sep 17 17:58:48.014897 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.014856    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6nlzz"]
Sep 17 17:58:48.016334 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.016318    2580 scope.go:117] "RemoveContainer" containerID="b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"
Sep 17 17:58:48.016589 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:58:48.016564    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b\": container with ID starting with b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b not found: ID does not exist" containerID="b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"
Sep 17 17:58:48.016649 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.016590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"} err="failed to get container status \"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b\": rpc error: code = NotFound desc = could not find container \"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b\": container with ID starting with b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b not found: ID does not exist"
Sep 17 17:58:48.016649 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.016609    2580 scope.go:117] "RemoveContainer" containerID="e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"
Sep 17 17:58:48.016876 ip-10-0-139-39 kubenswrapper[2580]: E0917 17:58:48.016859    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef\": container with ID starting with e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef not found: ID does not exist" containerID="e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"
Sep 17 17:58:48.016924 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.016885    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"} err="failed to get container status \"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef\": rpc error: code = NotFound desc = could not find container \"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef\": container with ID starting with e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef not found: ID does not exist"
Sep 17 17:58:48.016924 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.016902    2580 scope.go:117] "RemoveContainer" containerID="b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"
Sep 17 17:58:48.017193 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.017175    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b"} err="failed to get container status \"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b\": rpc error: code = NotFound desc = could not find container \"b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b\": container with ID starting with b6448909c0cd6de25af41f6d35d599aa180b872a308b1b6fd1f03b14f1c5b91b not found: ID does not exist"
Sep 17 17:58:48.017248 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.017195    2580 scope.go:117] "RemoveContainer" containerID="e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"
Sep 17 17:58:48.017442 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.017424    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef"} err="failed to get container status \"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef\": rpc error: code = NotFound desc = could not find container \"e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef\": container with ID starting with e28d4f05b78a4b71f4818b343b58346638d7528988d639f105e3dffffe95ddef not found: ID does not exist"
Sep 17 17:58:48.025495 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.025475    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"]
Sep 17 17:58:48.029663 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.029644    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s"]
Sep 17 17:58:48.046417 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.046399    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" path="/var/lib/kubelet/pods/c951a7ff-e40f-4b17-a313-2d961e94eca6/volumes"
Sep 17 17:58:48.046822 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.046810    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" path="/var/lib/kubelet/pods/da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad/volumes"
Sep 17 17:58:48.079691 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.079673    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.079691 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.079692    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.079691 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.079700    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.079849 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.079707    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c951a7ff-e40f-4b17-a313-2d961e94eca6-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.079849 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.079715    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c951a7ff-e40f-4b17-a313-2d961e94eca6-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.079849 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.079723    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrks2\" (UniqueName: \"kubernetes.io/projected/c951a7ff-e40f-4b17-a313-2d961e94eca6-kube-api-access-zrks2\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.459813 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:58:48.459782    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-trs9s" podUID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:59:35.408973 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:35.408934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:35.499935 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:35.499900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:41.899321 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:41.899279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:44.036650 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.036618    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"]
Sep 17 17:59:44.037011 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.036999    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" containerName="main"
Sep 17 17:59:44.037055 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037012    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" containerName="main"
Sep 17 17:59:44.037055 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037024    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="main"
Sep 17 17:59:44.037055 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037030    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="main"
Sep 17 17:59:44.037055 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037040    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="llm-d-routing-sidecar"
Sep 17 17:59:44.037055 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037046    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="llm-d-routing-sidecar"
Sep 17 17:59:44.037244 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037121    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="llm-d-routing-sidecar"
Sep 17 17:59:44.037244 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037131    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c951a7ff-e40f-4b17-a313-2d961e94eca6" containerName="main"
Sep 17 17:59:44.037244 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.037139    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da866688-d33c-4c9c-b6b9-3d3d4ba7c0ad" containerName="main"
Sep 17 17:59:44.042656 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.042623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.046488 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.046461    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-f8vzn\""
Sep 17 17:59:44.046626 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.046453    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-m4lmp\""
Sep 17 17:59:44.046905 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.046872    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 17:59:44.047119 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.047072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 17:59:44.047290 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.046984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 17:59:44.050573 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.050550    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"]
Sep 17 17:59:44.064434 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.064407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"]
Sep 17 17:59:44.064601 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.064556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.068430 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.068404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"]
Sep 17 17:59:44.095255 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.095227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:44.195066 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.195217 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zns\" (UniqueName: \"kubernetes.io/projected/86b905f8-0981-4ad3-893e-ccf9f6c27e09-kube-api-access-r7zns\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.195217 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87870c09-a556-45c6-9cfa-6c8be6a7e269-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.195320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.195320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.195320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.195320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.195320 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.195513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdr69\" (UniqueName: \"kubernetes.io/projected/87870c09-a556-45c6-9cfa-6c8be6a7e269-kube-api-access-fdr69\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.195513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.195513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.195513 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.195450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.296429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.296429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.296429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.296429 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.296684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.296684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdr69\" (UniqueName: \"kubernetes.io/projected/87870c09-a556-45c6-9cfa-6c8be6a7e269-kube-api-access-fdr69\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.296684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.296684 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.296907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.296907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.296907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.296907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.296907 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zns\" (UniqueName: \"kubernetes.io/projected/86b905f8-0981-4ad3-893e-ccf9f6c27e09-kube-api-access-r7zns\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.297215 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.296916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87870c09-a556-45c6-9cfa-6c8be6a7e269-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.297272 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.297234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.297331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.297292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.297331 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.297299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.297445 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.297399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.298951 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.298926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.299038 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.299000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.299260 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.299241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.299306 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.299259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87870c09-a556-45c6-9cfa-6c8be6a7e269-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.305322 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.305300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zns\" (UniqueName: \"kubernetes.io/projected/86b905f8-0981-4ad3-893e-ccf9f6c27e09-kube-api-access-r7zns\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.305673 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.305655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdr69\" (UniqueName: \"kubernetes.io/projected/87870c09-a556-45c6-9cfa-6c8be6a7e269-kube-api-access-fdr69\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.359247 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.359222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:44.378393 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.378366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:44.397285 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.397237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:44.660832 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.660811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"]
Sep 17 17:59:44.662693 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:59:44.662656    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b905f8_0981_4ad3_893e_ccf9f6c27e09.slice/crio-f165ad5782019b584c3f891e26b00fa70a92d8d80077b5c708ebf9e784ed238f WatchSource:0}: Error finding container f165ad5782019b584c3f891e26b00fa70a92d8d80077b5c708ebf9e784ed238f: Status 404 returned error can't find the container with id f165ad5782019b584c3f891e26b00fa70a92d8d80077b5c708ebf9e784ed238f
Sep 17 17:59:44.674797 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:44.674775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"]
Sep 17 17:59:44.676770 ip-10-0-139-39 kubenswrapper[2580]: W0917 17:59:44.676750    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87870c09_a556_45c6_9cfa_6c8be6a7e269.slice/crio-f2fd1794ee6045e46975d610f9bcb87161daaf95398fef05d9b798f9bab635c6 WatchSource:0}: Error finding container f2fd1794ee6045e46975d610f9bcb87161daaf95398fef05d9b798f9bab635c6: Status 404 returned error can't find the container with id f2fd1794ee6045e46975d610f9bcb87161daaf95398fef05d9b798f9bab635c6
Sep 17 17:59:45.315752 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:45.315715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" event={"ID":"86b905f8-0981-4ad3-893e-ccf9f6c27e09","Type":"ContainerStarted","Data":"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"}
Sep 17 17:59:45.316183 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:45.315778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:45.316183 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:45.315804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" event={"ID":"86b905f8-0981-4ad3-893e-ccf9f6c27e09","Type":"ContainerStarted","Data":"f165ad5782019b584c3f891e26b00fa70a92d8d80077b5c708ebf9e784ed238f"}
Sep 17 17:59:45.317446 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:45.317423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" event={"ID":"87870c09-a556-45c6-9cfa-6c8be6a7e269","Type":"ContainerStarted","Data":"be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a"}
Sep 17 17:59:45.317561 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:45.317462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" event={"ID":"87870c09-a556-45c6-9cfa-6c8be6a7e269","Type":"ContainerStarted","Data":"f2fd1794ee6045e46975d610f9bcb87161daaf95398fef05d9b798f9bab635c6"}
Sep 17 17:59:45.371294 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:45.371254    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" podStartSLOduration=1.371233949 podStartE2EDuration="1.371233949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:45.369891379 +0000 UTC m=+1161.709284972" watchObservedRunningTime="2026-09-17 17:59:45.371233949 +0000 UTC m=+1161.710627540"
Sep 17 17:59:45.379272 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:45.379249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:46.328622 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:46.328576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" event={"ID":"86b905f8-0981-4ad3-893e-ccf9f6c27e09","Type":"ContainerStarted","Data":"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"}
Sep 17 17:59:46.353151 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:46.353104    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" podStartSLOduration=2.353073057 podStartE2EDuration="2.353073057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:46.351216708 +0000 UTC m=+1162.690610366" watchObservedRunningTime="2026-09-17 17:59:46.353073057 +0000 UTC m=+1162.692466649"
Sep 17 17:59:46.359411 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:46.359392    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:48.541405 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:48.541366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:48.645007 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:48.644977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:54.360458 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:54.360418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:54.373987 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:54.373963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:54.374296 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:54.374265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:54.378741 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:54.378720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:54.387036 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:54.387015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 17:59:54.391099 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:54.391073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:55.391809 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:55.391771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 17:59:55.676448 ip-10-0-139-39 kubenswrapper[2580]: I0917 17:59:55.676365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:18.556665 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.556628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:18.575381 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.575340    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"]
Sep 17 18:00:18.583742 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.583715    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"]
Sep 17 18:00:18.587647 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:00:18.587614    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:00:18.587647 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:18.587647 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:18.587647 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a" cmd=["/bin/sleep","15"]
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.587664    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" message="" err=<
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:00:18.587680    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" podUID="87870c09-a556-45c6-9cfa-6c8be6a7e269" containerName="main" containerID="cri-o://be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a"
Sep 17 18:00:18.587914 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.587711    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" podUID="87870c09-a556-45c6-9cfa-6c8be6a7e269" containerName="main" containerID="cri-o://be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a" gracePeriod=60
Sep 17 18:00:18.592379 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:00:18.592345    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:00:18.592379 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:18.592379 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:18.592379 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f" cmd=["/bin/sleep","15"]
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.592387    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" message="" err=<
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:00:18.592398    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="main" containerID="cri-o://76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"
Sep 17 18:00:18.592613 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.592431    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="main" containerID="cri-o://76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f" gracePeriod=60
Sep 17 18:00:18.654605 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.654575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:18.810008 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.809976    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="llm-d-routing-sidecar" containerID="cri-o://4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f" gracePeriod=60
Sep 17 18:00:18.943613 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.943590    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 18:00:18.962969 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.962930    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-model-cache\") pod \"87870c09-a556-45c6-9cfa-6c8be6a7e269\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") "
Sep 17 18:00:18.962969 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.962968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-tmp-dir\") pod \"87870c09-a556-45c6-9cfa-6c8be6a7e269\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") "
Sep 17 18:00:18.963196 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.962998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87870c09-a556-45c6-9cfa-6c8be6a7e269-tls-certs\") pod \"87870c09-a556-45c6-9cfa-6c8be6a7e269\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") "
Sep 17 18:00:18.963196 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.963017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-home\") pod \"87870c09-a556-45c6-9cfa-6c8be6a7e269\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") "
Sep 17 18:00:18.963196 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.963034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-dshm\") pod \"87870c09-a556-45c6-9cfa-6c8be6a7e269\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") "
Sep 17 18:00:18.963196 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.963065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdr69\" (UniqueName: \"kubernetes.io/projected/87870c09-a556-45c6-9cfa-6c8be6a7e269-kube-api-access-fdr69\") pod \"87870c09-a556-45c6-9cfa-6c8be6a7e269\" (UID: \"87870c09-a556-45c6-9cfa-6c8be6a7e269\") "
Sep 17 18:00:18.963402 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.963262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-model-cache" (OuterVolumeSpecName: "model-cache") pod "87870c09-a556-45c6-9cfa-6c8be6a7e269" (UID: "87870c09-a556-45c6-9cfa-6c8be6a7e269"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:18.963402 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.963272    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87870c09-a556-45c6-9cfa-6c8be6a7e269" (UID: "87870c09-a556-45c6-9cfa-6c8be6a7e269"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:18.963402 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.963383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-home" (OuterVolumeSpecName: "home") pod "87870c09-a556-45c6-9cfa-6c8be6a7e269" (UID: "87870c09-a556-45c6-9cfa-6c8be6a7e269"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:18.965626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.965592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87870c09-a556-45c6-9cfa-6c8be6a7e269-kube-api-access-fdr69" (OuterVolumeSpecName: "kube-api-access-fdr69") pod "87870c09-a556-45c6-9cfa-6c8be6a7e269" (UID: "87870c09-a556-45c6-9cfa-6c8be6a7e269"). InnerVolumeSpecName "kube-api-access-fdr69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:18.965741 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.965645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87870c09-a556-45c6-9cfa-6c8be6a7e269-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87870c09-a556-45c6-9cfa-6c8be6a7e269" (UID: "87870c09-a556-45c6-9cfa-6c8be6a7e269"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:18.965741 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:18.965651    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-dshm" (OuterVolumeSpecName: "dshm") pod "87870c09-a556-45c6-9cfa-6c8be6a7e269" (UID: "87870c09-a556-45c6-9cfa-6c8be6a7e269"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:19.064139 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.064074    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.064139 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.064109    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87870c09-a556-45c6-9cfa-6c8be6a7e269-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.064139 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.064121    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.064139 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.064128    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.064139 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.064137    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdr69\" (UniqueName: \"kubernetes.io/projected/87870c09-a556-45c6-9cfa-6c8be6a7e269-kube-api-access-fdr69\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.064382 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.064144    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87870c09-a556-45c6-9cfa-6c8be6a7e269-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.064382 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.064192    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 18:00:19.164409 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tls-certs\") pod \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") "
Sep 17 18:00:19.164537 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164437    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7zns\" (UniqueName: \"kubernetes.io/projected/86b905f8-0981-4ad3-893e-ccf9f6c27e09-kube-api-access-r7zns\") pod \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") "
Sep 17 18:00:19.164537 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-model-cache\") pod \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") "
Sep 17 18:00:19.164537 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-home\") pod \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") "
Sep 17 18:00:19.164537 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tmp-dir\") pod \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") "
Sep 17 18:00:19.164735 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-dshm\") pod \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\" (UID: \"86b905f8-0981-4ad3-893e-ccf9f6c27e09\") "
Sep 17 18:00:19.164786 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-model-cache" (OuterVolumeSpecName: "model-cache") pod "86b905f8-0981-4ad3-893e-ccf9f6c27e09" (UID: "86b905f8-0981-4ad3-893e-ccf9f6c27e09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:19.164786 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86b905f8-0981-4ad3-893e-ccf9f6c27e09" (UID: "86b905f8-0981-4ad3-893e-ccf9f6c27e09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:19.164786 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-home" (OuterVolumeSpecName: "home") pod "86b905f8-0981-4ad3-893e-ccf9f6c27e09" (UID: "86b905f8-0981-4ad3-893e-ccf9f6c27e09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:19.164967 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164951    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.165029 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164970    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.165029 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.164983    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.166576 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.166551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86b905f8-0981-4ad3-893e-ccf9f6c27e09" (UID: "86b905f8-0981-4ad3-893e-ccf9f6c27e09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:19.166576 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.166571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b905f8-0981-4ad3-893e-ccf9f6c27e09-kube-api-access-r7zns" (OuterVolumeSpecName: "kube-api-access-r7zns") pod "86b905f8-0981-4ad3-893e-ccf9f6c27e09" (UID: "86b905f8-0981-4ad3-893e-ccf9f6c27e09"). InnerVolumeSpecName "kube-api-access-r7zns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:19.166733 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.166552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-dshm" (OuterVolumeSpecName: "dshm") pod "86b905f8-0981-4ad3-893e-ccf9f6c27e09" (UID: "86b905f8-0981-4ad3-893e-ccf9f6c27e09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:19.265560 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.265535    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b905f8-0981-4ad3-893e-ccf9f6c27e09-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.265560 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.265560    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7zns\" (UniqueName: \"kubernetes.io/projected/86b905f8-0981-4ad3-893e-ccf9f6c27e09-kube-api-access-r7zns\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.265721 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.265570    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b905f8-0981-4ad3-893e-ccf9f6c27e09-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:00:19.508287 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.508253    2580 generic.go:356] "Generic (PLEG): container finished" podID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerID="76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f" exitCode=0
Sep 17 18:00:19.508287 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.508282    2580 generic.go:356] "Generic (PLEG): container finished" podID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerID="4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f" exitCode=0
Sep 17 18:00:19.508491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.508303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"
Sep 17 18:00:19.508491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.508348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" event={"ID":"86b905f8-0981-4ad3-893e-ccf9f6c27e09","Type":"ContainerDied","Data":"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"}
Sep 17 18:00:19.508491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.508401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" event={"ID":"86b905f8-0981-4ad3-893e-ccf9f6c27e09","Type":"ContainerDied","Data":"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"}
Sep 17 18:00:19.508491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.508426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd" event={"ID":"86b905f8-0981-4ad3-893e-ccf9f6c27e09","Type":"ContainerDied","Data":"f165ad5782019b584c3f891e26b00fa70a92d8d80077b5c708ebf9e784ed238f"}
Sep 17 18:00:19.508491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.508448    2580 scope.go:117] "RemoveContainer" containerID="76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"
Sep 17 18:00:19.509815 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.509790    2580 generic.go:356] "Generic (PLEG): container finished" podID="87870c09-a556-45c6-9cfa-6c8be6a7e269" containerID="be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a" exitCode=0
Sep 17 18:00:19.509902 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.509830    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"
Sep 17 18:00:19.509902 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.509845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" event={"ID":"87870c09-a556-45c6-9cfa-6c8be6a7e269","Type":"ContainerDied","Data":"be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a"}
Sep 17 18:00:19.509902 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.509896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9" event={"ID":"87870c09-a556-45c6-9cfa-6c8be6a7e269","Type":"ContainerDied","Data":"f2fd1794ee6045e46975d610f9bcb87161daaf95398fef05d9b798f9bab635c6"}
Sep 17 18:00:19.518063 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.518047    2580 scope.go:117] "RemoveContainer" containerID="4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"
Sep 17 18:00:19.527061 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.527044    2580 scope.go:117] "RemoveContainer" containerID="76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"
Sep 17 18:00:19.527397 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:00:19.527376    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f\": container with ID starting with 76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f not found: ID does not exist" containerID="76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"
Sep 17 18:00:19.527480 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.527401    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"} err="failed to get container status \"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f\": rpc error: code = NotFound desc = could not find container \"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f\": container with ID starting with 76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f not found: ID does not exist"
Sep 17 18:00:19.527480 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.527418    2580 scope.go:117] "RemoveContainer" containerID="4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"
Sep 17 18:00:19.527693 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:00:19.527675    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f\": container with ID starting with 4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f not found: ID does not exist" containerID="4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"
Sep 17 18:00:19.527746 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.527698    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"} err="failed to get container status \"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f\": rpc error: code = NotFound desc = could not find container \"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f\": container with ID starting with 4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f not found: ID does not exist"
Sep 17 18:00:19.527746 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.527713    2580 scope.go:117] "RemoveContainer" containerID="76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"
Sep 17 18:00:19.527966 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.527950    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f"} err="failed to get container status \"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f\": rpc error: code = NotFound desc = could not find container \"76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f\": container with ID starting with 76c5534ed13f2a4915ec44f9654e4441e8e4c490984664fa11710d3dfafd155f not found: ID does not exist"
Sep 17 18:00:19.528027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.527967    2580 scope.go:117] "RemoveContainer" containerID="4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"
Sep 17 18:00:19.528214 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.528196    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f"} err="failed to get container status \"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f\": rpc error: code = NotFound desc = could not find container \"4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f\": container with ID starting with 4c447dc128e55a8a12da274701effe9def1f0a3ab4c9c1bc38dcd757f92e9d6f not found: ID does not exist"
Sep 17 18:00:19.528287 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.528214    2580 scope.go:117] "RemoveContainer" containerID="be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a"
Sep 17 18:00:19.535881 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.535866    2580 scope.go:117] "RemoveContainer" containerID="be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a"
Sep 17 18:00:19.536170 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:00:19.536143    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a\": container with ID starting with be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a not found: ID does not exist" containerID="be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a"
Sep 17 18:00:19.536223 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.536174    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a"} err="failed to get container status \"be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a\": rpc error: code = NotFound desc = could not find container \"be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a\": container with ID starting with be0b7496c13537d21377d9bc85f0138c8674eab1ed7f668b28097c0f7ff61a6a not found: ID does not exist"
Sep 17 18:00:19.597027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.597001    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"]
Sep 17 18:00:19.600192 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.600174    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bt6w9"]
Sep 17 18:00:19.611237 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.611219    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"]
Sep 17 18:00:19.624540 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:19.624514    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-x79vd"]
Sep 17 18:00:20.045698 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:20.045668    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" path="/var/lib/kubelet/pods/86b905f8-0981-4ad3-893e-ccf9f6c27e09/volumes"
Sep 17 18:00:20.046247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:20.046223    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87870c09-a556-45c6-9cfa-6c8be6a7e269" path="/var/lib/kubelet/pods/87870c09-a556-45c6-9cfa-6c8be6a7e269/volumes"
Sep 17 18:00:25.097611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:25.097585    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:00:25.104598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:25.104569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:00:46.930421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:46.930381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:47.027252 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:00:47.027190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:00.384266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:00.384223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:00.482073 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:00.482034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:08.395430 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:08.395392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:08.494574 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:08.494540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:12.008048 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:12.008012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:13.383198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383155    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"]
Sep 17 18:01:13.383857 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383826    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="llm-d-routing-sidecar"
Sep 17 18:01:13.383857 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383857    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="llm-d-routing-sidecar"
Sep 17 18:01:13.384064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87870c09-a556-45c6-9cfa-6c8be6a7e269" containerName="main"
Sep 17 18:01:13.384064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383881    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87870c09-a556-45c6-9cfa-6c8be6a7e269" containerName="main"
Sep 17 18:01:13.384064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383893    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="main"
Sep 17 18:01:13.384064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383898    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="main"
Sep 17 18:01:13.384064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383966    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87870c09-a556-45c6-9cfa-6c8be6a7e269" containerName="main"
Sep 17 18:01:13.384064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383977    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="main"
Sep 17 18:01:13.384064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.383984    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b905f8-0981-4ad3-893e-ccf9f6c27e09" containerName="llm-d-routing-sidecar"
Sep 17 18:01:13.387424 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.387399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.390299 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.390264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:13.390447 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.390326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 18:01:13.391413 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.391386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 18:01:13.391413 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.391404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:13.391596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.391418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 18:01:13.391596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.391420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qbz5s\""
Sep 17 18:01:13.403928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.403890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"]
Sep 17 18:01:13.428336 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.428294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:13.460832 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.460794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-home\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.461025 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.460837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55881c12-2b05-486b-b514-9e94e597157d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.461025 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.460873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-dshm\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.461025 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.460898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.461025 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.460936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.461025 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.461014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-model-cache\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.461314 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.461035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvxx\" (UniqueName: \"kubernetes.io/projected/55881c12-2b05-486b-b514-9e94e597157d-kube-api-access-qfvxx\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.461314 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.461075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55881c12-2b05-486b-b514-9e94e597157d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562161 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-model-cache\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvxx\" (UniqueName: \"kubernetes.io/projected/55881c12-2b05-486b-b514-9e94e597157d-kube-api-access-qfvxx\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55881c12-2b05-486b-b514-9e94e597157d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-home\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55881c12-2b05-486b-b514-9e94e597157d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-dshm\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562676 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562676 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-model-cache\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562676 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562855 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-home\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.562855 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.562797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.563132 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.563080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55881c12-2b05-486b-b514-9e94e597157d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.564754 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.564723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-dshm\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.564948 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.564930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55881c12-2b05-486b-b514-9e94e597157d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.571893 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.571866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvxx\" (UniqueName: \"kubernetes.io/projected/55881c12-2b05-486b-b514-9e94e597157d-kube-api-access-qfvxx\") pod \"tokenizer-clean-path-test-kserve-759bc445b5-v4hm8\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.618583 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.618549    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"]
Sep 17 18:01:13.623296 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.623270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.636442 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.636370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"]
Sep 17 18:01:13.663464 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.663411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:13.700054 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.700019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:13.763788 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.763746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.763959 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.763809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.763959 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.763880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db6478c-de6d-4508-9416-7a14675a027a-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.763959 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.763904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db6478c-de6d-4508-9416-7a14675a027a-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.864537 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.864504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.864724 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.864563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db6478c-de6d-4508-9416-7a14675a027a-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.864724 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.864584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db6478c-de6d-4508-9416-7a14675a027a-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.864724 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.864634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.864964 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.864946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.865012 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.864998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.865301 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.865278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db6478c-de6d-4508-9416-7a14675a027a-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.867331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.867310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db6478c-de6d-4508-9416-7a14675a027a-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.934000 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.933950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:01:13.982193 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:13.982156    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"]
Sep 17 18:01:13.982686 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:01:13.982658    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55881c12_2b05_486b_b514_9e94e597157d.slice/crio-22fd40f210408ae67a457c2ca40d45f6aa61a812f85ec55067a72697cf97bb83 WatchSource:0}: Error finding container 22fd40f210408ae67a457c2ca40d45f6aa61a812f85ec55067a72697cf97bb83: Status 404 returned error can't find the container with id 22fd40f210408ae67a457c2ca40d45f6aa61a812f85ec55067a72697cf97bb83
Sep 17 18:01:14.244348 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:14.244323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"]
Sep 17 18:01:14.246401 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:01:14.246373    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db6478c_de6d_4508_9416_7a14675a027a.slice/crio-bbfcfc9302c37dbf73fe5ee9465b39947c545b3c299c5c4703e6a6571169e580 WatchSource:0}: Error finding container bbfcfc9302c37dbf73fe5ee9465b39947c545b3c299c5c4703e6a6571169e580: Status 404 returned error can't find the container with id bbfcfc9302c37dbf73fe5ee9465b39947c545b3c299c5c4703e6a6571169e580
Sep 17 18:01:14.846576 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:14.846526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" event={"ID":"55881c12-2b05-486b-b514-9e94e597157d","Type":"ContainerStarted","Data":"ac96af87dbdd5413d5de7ad6a6032fb98019ef6e7b540a7031196addaedf0fae"}
Sep 17 18:01:14.847351 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:14.846610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" event={"ID":"55881c12-2b05-486b-b514-9e94e597157d","Type":"ContainerStarted","Data":"22fd40f210408ae67a457c2ca40d45f6aa61a812f85ec55067a72697cf97bb83"}
Sep 17 18:01:14.848749 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:14.848707    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" event={"ID":"0db6478c-de6d-4508-9416-7a14675a027a","Type":"ContainerStarted","Data":"7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24"}
Sep 17 18:01:14.848878 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:14.848783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" event={"ID":"0db6478c-de6d-4508-9416-7a14675a027a","Type":"ContainerStarted","Data":"bbfcfc9302c37dbf73fe5ee9465b39947c545b3c299c5c4703e6a6571169e580"}
Sep 17 18:01:16.089978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:16.089938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:16.189681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:16.189635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:16.864530 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:16.864498    2580 generic.go:356] "Generic (PLEG): container finished" podID="55881c12-2b05-486b-b514-9e94e597157d" containerID="ac96af87dbdd5413d5de7ad6a6032fb98019ef6e7b540a7031196addaedf0fae" exitCode=0
Sep 17 18:01:16.864680 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:16.864576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" event={"ID":"55881c12-2b05-486b-b514-9e94e597157d","Type":"ContainerDied","Data":"ac96af87dbdd5413d5de7ad6a6032fb98019ef6e7b540a7031196addaedf0fae"}
Sep 17 18:01:17.871696 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:17.871664    2580 generic.go:356] "Generic (PLEG): container finished" podID="0db6478c-de6d-4508-9416-7a14675a027a" containerID="7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24" exitCode=0
Sep 17 18:01:17.872200 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:17.871742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" event={"ID":"0db6478c-de6d-4508-9416-7a14675a027a","Type":"ContainerDied","Data":"7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24"}
Sep 17 18:01:17.873844 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:17.873820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" event={"ID":"55881c12-2b05-486b-b514-9e94e597157d","Type":"ContainerStarted","Data":"bd7cc47466df6267086a44de83098e12ae0da680936e0e84c120795cf1675ff6"}
Sep 17 18:01:17.914155 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:17.914114    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" podStartSLOduration=2.318279066 podStartE2EDuration="4.914099257s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.595820191s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:17.911503773 +0000 UTC m=+1254.250897390" watchObservedRunningTime="2026-09-17 18:01:17.914099257 +0000 UTC m=+1254.253492859"
Sep 17 18:01:18.701175 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:18.701135    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:21.231377 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:21.231339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:23.700789 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:23.700749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:23.719973 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:23.719943    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:23.932640 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:23.932611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:01:25.211198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:25.211139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:36.807907 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:36.807864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:36.903419 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:36.903305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:43.196708 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:43.196669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:43.398124 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:43.398066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:43.563137 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:43.562259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:49.537079 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:49.537037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:49.715470 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:49.715437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:54.729855 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:54.729802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:54.801635 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:01:54.801601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:06.203469 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:06.203433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" event={"ID":"0db6478c-de6d-4508-9416-7a14675a027a","Type":"ContainerStarted","Data":"f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0"}
Sep 17 18:02:06.203882 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:06.203636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:02:06.224037 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:06.223993    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" podStartSLOduration=3.071318269 podStartE2EDuration="53.223980106s" totalImagesPullingTime="47.570327051s" totalInitContainerRuntime="2.582334786s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:06.222870469 +0000 UTC m=+1302.562264062" watchObservedRunningTime="2026-09-17 18:02:06.223980106 +0000 UTC m=+1302.563373697"
Sep 17 18:02:08.019158 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:08.019121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:37.220759 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:37.220728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:02:38.694945 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.694910    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw_0db6478c-de6d-4508-9416-7a14675a027a/main/0.log"
Sep 17 18:02:38.878618 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.878586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:38.885402 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.885367    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"]
Sep 17 18:02:38.885644 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.885618    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" podUID="0db6478c-de6d-4508-9416-7a14675a027a" containerName="main" containerID="cri-o://f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0" gracePeriod=30
Sep 17 18:02:38.889038 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.889009    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"]
Sep 17 18:02:38.899220 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:02:38.899182    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:02:38.899220 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:38.899220 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:38.899220 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="bd7cc47466df6267086a44de83098e12ae0da680936e0e84c120795cf1675ff6" cmd=["/bin/sleep","15"]
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.899238    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" message="" err=<
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:02:38.899249    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="main" containerID="cri-o://bd7cc47466df6267086a44de83098e12ae0da680936e0e84c120795cf1675ff6"
Sep 17 18:02:38.899501 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.899292    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="main" containerID="cri-o://bd7cc47466df6267086a44de83098e12ae0da680936e0e84c120795cf1675ff6" gracePeriod=60
Sep 17 18:02:38.916783 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.916760    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 18:02:38.956619 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:38.956546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:39.400298 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.400269    2580 generic.go:356] "Generic (PLEG): container finished" podID="55881c12-2b05-486b-b514-9e94e597157d" containerID="bd7cc47466df6267086a44de83098e12ae0da680936e0e84c120795cf1675ff6" exitCode=0
Sep 17 18:02:39.400460 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.400339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" event={"ID":"55881c12-2b05-486b-b514-9e94e597157d","Type":"ContainerDied","Data":"bd7cc47466df6267086a44de83098e12ae0da680936e0e84c120795cf1675ff6"}
Sep 17 18:02:39.576376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.576311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:02:39.625170 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.625138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-kserve-provision-location\") pod \"0db6478c-de6d-4508-9416-7a14675a027a\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") "
Sep 17 18:02:39.625333 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.625189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db6478c-de6d-4508-9416-7a14675a027a-cabundle-cert\") pod \"0db6478c-de6d-4508-9416-7a14675a027a\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") "
Sep 17 18:02:39.625333 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.625225    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db6478c-de6d-4508-9416-7a14675a027a-tls-certs\") pod \"0db6478c-de6d-4508-9416-7a14675a027a\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") "
Sep 17 18:02:39.625333 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.625311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-tmp-dir\") pod \"0db6478c-de6d-4508-9416-7a14675a027a\" (UID: \"0db6478c-de6d-4508-9416-7a14675a027a\") "
Sep 17 18:02:39.625926 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.625739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db6478c-de6d-4508-9416-7a14675a027a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0db6478c-de6d-4508-9416-7a14675a027a" (UID: "0db6478c-de6d-4508-9416-7a14675a027a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:39.625926 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.625917    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0db6478c-de6d-4508-9416-7a14675a027a" (UID: "0db6478c-de6d-4508-9416-7a14675a027a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.628564 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.628529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db6478c-de6d-4508-9416-7a14675a027a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0db6478c-de6d-4508-9416-7a14675a027a" (UID: "0db6478c-de6d-4508-9416-7a14675a027a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:39.648403 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.648379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0db6478c-de6d-4508-9416-7a14675a027a" (UID: "0db6478c-de6d-4508-9416-7a14675a027a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.704489 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.704470    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:02:39.726746 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726716    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-tmp-dir\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.726838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-dshm\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.726838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfvxx\" (UniqueName: \"kubernetes.io/projected/55881c12-2b05-486b-b514-9e94e597157d-kube-api-access-qfvxx\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.726838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-home\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.726957 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-model-cache\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.726957 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-kserve-provision-location\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.726957 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726902    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55881c12-2b05-486b-b514-9e94e597157d-tls-certs\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.726957 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.726925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55881c12-2b05-486b-b514-9e94e597157d-cabundle-cert\") pod \"55881c12-2b05-486b-b514-9e94e597157d\" (UID: \"55881c12-2b05-486b-b514-9e94e597157d\") "
Sep 17 18:02:39.727196 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727024    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.727196 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-home" (OuterVolumeSpecName: "home") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.727196 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-model-cache" (OuterVolumeSpecName: "model-cache") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.727364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727286    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.727364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727304    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.727364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727317    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.727364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727331    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db6478c-de6d-4508-9416-7a14675a027a-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.727364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727344    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db6478c-de6d-4508-9416-7a14675a027a-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.727364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727357    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db6478c-de6d-4508-9416-7a14675a027a-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.727674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727373    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.727674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.727352    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55881c12-2b05-486b-b514-9e94e597157d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:39.728932 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.728913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-dshm" (OuterVolumeSpecName: "dshm") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.729030 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.728974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55881c12-2b05-486b-b514-9e94e597157d-kube-api-access-qfvxx" (OuterVolumeSpecName: "kube-api-access-qfvxx") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "kube-api-access-qfvxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:39.729104 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.729032    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55881c12-2b05-486b-b514-9e94e597157d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:39.749274 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.749238    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55881c12-2b05-486b-b514-9e94e597157d" (UID: "55881c12-2b05-486b-b514-9e94e597157d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.828066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.828003    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.828066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.828028    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfvxx\" (UniqueName: \"kubernetes.io/projected/55881c12-2b05-486b-b514-9e94e597157d-kube-api-access-qfvxx\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.828066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.828040    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55881c12-2b05-486b-b514-9e94e597157d-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.828066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.828049    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55881c12-2b05-486b-b514-9e94e597157d-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.828066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:39.828058    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55881c12-2b05-486b-b514-9e94e597157d-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:02:40.407628 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.407589    2580 generic.go:356] "Generic (PLEG): container finished" podID="0db6478c-de6d-4508-9416-7a14675a027a" containerID="f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0" exitCode=0
Sep 17 18:02:40.407784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.407642    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"
Sep 17 18:02:40.407784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.407643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" event={"ID":"0db6478c-de6d-4508-9416-7a14675a027a","Type":"ContainerDied","Data":"f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0"}
Sep 17 18:02:40.407784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.407702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw" event={"ID":"0db6478c-de6d-4508-9416-7a14675a027a","Type":"ContainerDied","Data":"bbfcfc9302c37dbf73fe5ee9465b39947c545b3c299c5c4703e6a6571169e580"}
Sep 17 18:02:40.407784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.407727    2580 scope.go:117] "RemoveContainer" containerID="f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0"
Sep 17 18:02:40.409789 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.409707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"
Sep 17 18:02:40.409789 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.409731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8" event={"ID":"55881c12-2b05-486b-b514-9e94e597157d","Type":"ContainerDied","Data":"22fd40f210408ae67a457c2ca40d45f6aa61a812f85ec55067a72697cf97bb83"}
Sep 17 18:02:40.418011 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.417064    2580 scope.go:117] "RemoveContainer" containerID="7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24"
Sep 17 18:02:40.425294 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.425274    2580 scope.go:117] "RemoveContainer" containerID="f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0"
Sep 17 18:02:40.425597 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:02:40.425574    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0\": container with ID starting with f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0 not found: ID does not exist" containerID="f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0"
Sep 17 18:02:40.425688 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.425612    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0"} err="failed to get container status \"f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0\": rpc error: code = NotFound desc = could not find container \"f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0\": container with ID starting with f2927a70c855c8abad9e1b09aa617b7d071a7dd21ff37ccbe3ff2c2b2cc43ac0 not found: ID does not exist"
Sep 17 18:02:40.425688 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.425637    2580 scope.go:117] "RemoveContainer" containerID="7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24"
Sep 17 18:02:40.425915 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:02:40.425887    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24\": container with ID starting with 7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24 not found: ID does not exist" containerID="7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24"
Sep 17 18:02:40.425970 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.425947    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24"} err="failed to get container status \"7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24\": rpc error: code = NotFound desc = could not find container \"7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24\": container with ID starting with 7b404586b7e20317eb8bd8ed3ae65a14197673371a50f57d9cfcfe32b62c6a24 not found: ID does not exist"
Sep 17 18:02:40.426023 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.425975    2580 scope.go:117] "RemoveContainer" containerID="bd7cc47466df6267086a44de83098e12ae0da680936e0e84c120795cf1675ff6"
Sep 17 18:02:40.428596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.428576    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"]
Sep 17 18:02:40.433598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.433576    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7fdbcfc8cb-8qwqw"]
Sep 17 18:02:40.434490 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.434475    2580 scope.go:117] "RemoveContainer" containerID="ac96af87dbdd5413d5de7ad6a6032fb98019ef6e7b540a7031196addaedf0fae"
Sep 17 18:02:40.447940 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.447920    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"]
Sep 17 18:02:40.454927 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:40.454907    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-759bc445b5-v4hm8"]
Sep 17 18:02:42.046628 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:42.046593    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db6478c-de6d-4508-9416-7a14675a027a" path="/var/lib/kubelet/pods/0db6478c-de6d-4508-9416-7a14675a027a/volumes"
Sep 17 18:02:42.047064 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:42.047048    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55881c12-2b05-486b-b514-9e94e597157d" path="/var/lib/kubelet/pods/55881c12-2b05-486b-b514-9e94e597157d/volumes"
Sep 17 18:02:44.183700 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:02:44.183656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:29.024671 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:29.024621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:29.122383 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:29.122344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:35.828307 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:35.828259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:42.187868 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.187830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:42.296908 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.296878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:42.399519 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399489    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"]
Sep 17 18:03:42.399944 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399926    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="storage-initializer"
Sep 17 18:03:42.399944 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399945    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="storage-initializer"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db6478c-de6d-4508-9416-7a14675a027a" containerName="main"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399958    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db6478c-de6d-4508-9416-7a14675a027a" containerName="main"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399972    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db6478c-de6d-4508-9416-7a14675a027a" containerName="storage-initializer"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399979    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db6478c-de6d-4508-9416-7a14675a027a" containerName="storage-initializer"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399990    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="main"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.399995    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="main"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.400051    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db6478c-de6d-4508-9416-7a14675a027a" containerName="main"
Sep 17 18:03:42.400093 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.400060    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="55881c12-2b05-486b-b514-9e94e597157d" containerName="main"
Sep 17 18:03:42.404707 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.404688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.407596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.407563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 18:03:42.407708 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.407658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:03:42.408281 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.408255    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:03:42.408281 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.408256    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:03:42.408954 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.408919    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-2c8nj\""
Sep 17 18:03:42.408954 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.408934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:03:42.415775 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.415751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"]
Sep 17 18:03:42.449611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.449611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqfl\" (UniqueName: \"kubernetes.io/projected/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kube-api-access-bkqfl\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.449611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.449786 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-home\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.449786 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.449786 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-model-cache\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.449904 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.449904 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.449844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-dshm\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550454 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-model-cache\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550573 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550635 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-dshm\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550694 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550694 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqfl\" (UniqueName: \"kubernetes.io/projected/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kube-api-access-bkqfl\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550822 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550822 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-model-cache\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550932 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-home\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550932 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.550932 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.550878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.551127 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.551067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.551194 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.551155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-home\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.551344 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.551312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.552943 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.552916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-dshm\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.553380 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.553365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.559482 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.559464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqfl\" (UniqueName: \"kubernetes.io/projected/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kube-api-access-bkqfl\") pod \"tokenizer-migration-path-test-kserve-69665bb668-9r92t\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.655813 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.655787    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"]
Sep 17 18:03:42.660404 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.660385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.676131 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.676100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"]
Sep 17 18:03:42.716115 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.716037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:42.753039 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.753012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.753172 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.753042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.753172 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.753074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.753428 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.753184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.854574 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.854542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.854697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.854576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.854697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.854605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.854697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.854648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.854996 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.854979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.855140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.855114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.855406 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.855386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.856926 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.856905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:42.972167 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:42.972102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:43.001999 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.001976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"]
Sep 17 18:03:43.004158 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:03:43.004133    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09baff7_f8aa_44dd_ba95_6263b3d56a9d.slice/crio-dff8232a4f6ecb84163621975712b153d88c486ba679e434a6468f943c573e3a WatchSource:0}: Error finding container dff8232a4f6ecb84163621975712b153d88c486ba679e434a6468f943c573e3a: Status 404 returned error can't find the container with id dff8232a4f6ecb84163621975712b153d88c486ba679e434a6468f943c573e3a
Sep 17 18:03:43.255897 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.255818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"]
Sep 17 18:03:43.258642 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:03:43.258607    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc227aac9_2c77_4501_9c1c_1dcd24bfd36a.slice/crio-83261fb85bcbeedf2752a19f49284ca1edd078965d93fc77b17fae7f0890b925 WatchSource:0}: Error finding container 83261fb85bcbeedf2752a19f49284ca1edd078965d93fc77b17fae7f0890b925: Status 404 returned error can't find the container with id 83261fb85bcbeedf2752a19f49284ca1edd078965d93fc77b17fae7f0890b925
Sep 17 18:03:43.515901 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.515797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:43.592972 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.592941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:43.770577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.770475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" event={"ID":"c227aac9-2c77-4501-9c1c-1dcd24bfd36a","Type":"ContainerStarted","Data":"4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f"}
Sep 17 18:03:43.770577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.770540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" event={"ID":"c227aac9-2c77-4501-9c1c-1dcd24bfd36a","Type":"ContainerStarted","Data":"83261fb85bcbeedf2752a19f49284ca1edd078965d93fc77b17fae7f0890b925"}
Sep 17 18:03:43.772434 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.772407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" event={"ID":"f09baff7-f8aa-44dd-ba95-6263b3d56a9d","Type":"ContainerStarted","Data":"e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab"}
Sep 17 18:03:43.772575 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:43.772448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" event={"ID":"f09baff7-f8aa-44dd-ba95-6263b3d56a9d","Type":"ContainerStarted","Data":"dff8232a4f6ecb84163621975712b153d88c486ba679e434a6468f943c573e3a"}
Sep 17 18:03:45.788317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:45.788236    2580 generic.go:356] "Generic (PLEG): container finished" podID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerID="e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab" exitCode=0
Sep 17 18:03:45.788317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:45.788291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" event={"ID":"f09baff7-f8aa-44dd-ba95-6263b3d56a9d","Type":"ContainerDied","Data":"e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab"}
Sep 17 18:03:46.794767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:46.794732    2580 generic.go:356] "Generic (PLEG): container finished" podID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerID="4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f" exitCode=0
Sep 17 18:03:46.795209 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:46.794807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" event={"ID":"c227aac9-2c77-4501-9c1c-1dcd24bfd36a","Type":"ContainerDied","Data":"4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f"}
Sep 17 18:03:46.796858 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:46.796834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" event={"ID":"f09baff7-f8aa-44dd-ba95-6263b3d56a9d","Type":"ContainerStarted","Data":"c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d"}
Sep 17 18:03:46.889549 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:46.889505    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" podStartSLOduration=2.437127417 podStartE2EDuration="4.88949169s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.452364273s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:46.885171816 +0000 UTC m=+1403.224565408" watchObservedRunningTime="2026-09-17 18:03:46.88949169 +0000 UTC m=+1403.228885282"
Sep 17 18:03:47.716492 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:47.716457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:47.804706 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:47.804673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" event={"ID":"c227aac9-2c77-4501-9c1c-1dcd24bfd36a","Type":"ContainerStarted","Data":"237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0"}
Sep 17 18:03:47.805113 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:47.805040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:03:47.828378 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:47.828335    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" podStartSLOduration=3.344051111 podStartE2EDuration="5.828321322s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.484270211s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:47.827803867 +0000 UTC m=+1404.167197462" watchObservedRunningTime="2026-09-17 18:03:47.828321322 +0000 UTC m=+1404.167714912"
Sep 17 18:03:52.717219 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:52.717187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:52.729885 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:52.729862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:52.847724 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:52.847695    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:03:53.766265 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:53.766227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:57.363817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:57.363774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:57.443105 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:03:57.443049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:18.821883 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:18.821793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:04:20.452980 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.452948    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr_c227aac9-2c77-4501-9c1c-1dcd24bfd36a/main/0.log"
Sep 17 18:04:20.777273 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.777175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:20.815466 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.815430    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"]
Sep 17 18:04:20.826048 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:04:20.826015    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:04:20.826048 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:20.826048 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:20.826048 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d" cmd=["/bin/sleep","15"]
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.826062    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" message="" err=<
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:04:20.826079    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="main" containerID="cri-o://c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d"
Sep 17 18:04:20.826387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.826126    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="main" containerID="cri-o://c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d" gracePeriod=60
Sep 17 18:04:20.835939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.835904    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"]
Sep 17 18:04:20.836165 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.836142    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" podUID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerName="main" containerID="cri-o://237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0" gracePeriod=30
Sep 17 18:04:20.836848 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.836819    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:04:20.892133 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:20.892105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:21.189511 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.189484    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:04:21.266592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kserve-provision-location\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.266790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tmp-dir\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.266790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-dshm\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.266790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-home\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.266790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-model-cache\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.266790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266790    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkqfl\" (UniqueName: \"kubernetes.io/projected/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kube-api-access-bkqfl\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.267067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266821    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tls-certs\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.267067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-cabundle-cert\") pod \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\" (UID: \"f09baff7-f8aa-44dd-ba95-6263b3d56a9d\") "
Sep 17 18:04:21.267067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.266883    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.267067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.267035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.267307 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.267121    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.267307 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.267137    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.267656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.267615    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-home" (OuterVolumeSpecName: "home") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.267656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.267631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:21.269622 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.269593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kube-api-access-bkqfl" (OuterVolumeSpecName: "kube-api-access-bkqfl") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "kube-api-access-bkqfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:21.270075 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.270048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-dshm" (OuterVolumeSpecName: "dshm") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.270376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.270348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:21.291139 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.291113    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f09baff7-f8aa-44dd-ba95-6263b3d56a9d" (UID: "f09baff7-f8aa-44dd-ba95-6263b3d56a9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.368269 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.368243    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.368269 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.368269    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.368427 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.368279    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.368427 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.368287    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqfl\" (UniqueName: \"kubernetes.io/projected/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-kube-api-access-bkqfl\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.368427 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.368295    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.368427 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.368304    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f09baff7-f8aa-44dd-ba95-6263b3d56a9d-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.480503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.480482    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:04:21.570146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.570045    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-kserve-provision-location\") pod \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") "
Sep 17 18:04:21.570146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.570116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tls-certs\") pod \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") "
Sep 17 18:04:21.570344 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.570154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tmp-dir\") pod \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") "
Sep 17 18:04:21.570344 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.570241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-cabundle-cert\") pod \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\" (UID: \"c227aac9-2c77-4501-9c1c-1dcd24bfd36a\") "
Sep 17 18:04:21.570647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.570613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c227aac9-2c77-4501-9c1c-1dcd24bfd36a" (UID: "c227aac9-2c77-4501-9c1c-1dcd24bfd36a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.570779 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.570694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c227aac9-2c77-4501-9c1c-1dcd24bfd36a" (UID: "c227aac9-2c77-4501-9c1c-1dcd24bfd36a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:21.572717 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.572697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c227aac9-2c77-4501-9c1c-1dcd24bfd36a" (UID: "c227aac9-2c77-4501-9c1c-1dcd24bfd36a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:21.592012 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.591986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c227aac9-2c77-4501-9c1c-1dcd24bfd36a" (UID: "c227aac9-2c77-4501-9c1c-1dcd24bfd36a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:21.670820 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.670797    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.670820 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.670819    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.670953 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.670830    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:21.670953 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:21.670839    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c227aac9-2c77-4501-9c1c-1dcd24bfd36a-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:04:22.010696 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.010667    2580 generic.go:356] "Generic (PLEG): container finished" podID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerID="237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0" exitCode=0
Sep 17 18:04:22.010830 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.010724    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"
Sep 17 18:04:22.010830 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.010759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" event={"ID":"c227aac9-2c77-4501-9c1c-1dcd24bfd36a","Type":"ContainerDied","Data":"237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0"}
Sep 17 18:04:22.010830 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.010824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr" event={"ID":"c227aac9-2c77-4501-9c1c-1dcd24bfd36a","Type":"ContainerDied","Data":"83261fb85bcbeedf2752a19f49284ca1edd078965d93fc77b17fae7f0890b925"}
Sep 17 18:04:22.010989 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.010856    2580 scope.go:117] "RemoveContainer" containerID="237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0"
Sep 17 18:04:22.016437 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.016411    2580 generic.go:356] "Generic (PLEG): container finished" podID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerID="c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d" exitCode=0
Sep 17 18:04:22.016644 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.016447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" event={"ID":"f09baff7-f8aa-44dd-ba95-6263b3d56a9d","Type":"ContainerDied","Data":"c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d"}
Sep 17 18:04:22.016785 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.016665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t" event={"ID":"f09baff7-f8aa-44dd-ba95-6263b3d56a9d","Type":"ContainerDied","Data":"dff8232a4f6ecb84163621975712b153d88c486ba679e434a6468f943c573e3a"}
Sep 17 18:04:22.016785 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.016472    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"
Sep 17 18:04:22.025109 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.025059    2580 scope.go:117] "RemoveContainer" containerID="4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f"
Sep 17 18:04:22.033697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.033678    2580 scope.go:117] "RemoveContainer" containerID="237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0"
Sep 17 18:04:22.033991 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:04:22.033959    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0\": container with ID starting with 237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0 not found: ID does not exist" containerID="237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0"
Sep 17 18:04:22.034079 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.034000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0"} err="failed to get container status \"237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0\": rpc error: code = NotFound desc = could not find container \"237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0\": container with ID starting with 237223899c075439946bf31300ad12256c4cbf70440261bd7247b4c3e36953b0 not found: ID does not exist"
Sep 17 18:04:22.034079 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.034023    2580 scope.go:117] "RemoveContainer" containerID="4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f"
Sep 17 18:04:22.034310 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:04:22.034294    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f\": container with ID starting with 4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f not found: ID does not exist" containerID="4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f"
Sep 17 18:04:22.034350 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.034317    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f"} err="failed to get container status \"4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f\": rpc error: code = NotFound desc = could not find container \"4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f\": container with ID starting with 4e5190c77f44b46c18bd4a7ebf8a7b2ed41d86a3fb1e744adefc8b944552ee7f not found: ID does not exist"
Sep 17 18:04:22.034350 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.034331    2580 scope.go:117] "RemoveContainer" containerID="c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d"
Sep 17 18:04:22.044308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.044174    2580 scope.go:117] "RemoveContainer" containerID="e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab"
Sep 17 18:04:22.046564 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.046541    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"]
Sep 17 18:04:22.046653 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.046577    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-554996ffcd-tf8pr"]
Sep 17 18:04:22.051940 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.051924    2580 scope.go:117] "RemoveContainer" containerID="c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d"
Sep 17 18:04:22.052210 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:04:22.052192    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d\": container with ID starting with c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d not found: ID does not exist" containerID="c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d"
Sep 17 18:04:22.052270 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.052216    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d"} err="failed to get container status \"c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d\": rpc error: code = NotFound desc = could not find container \"c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d\": container with ID starting with c21e070bd2d301364852cc62935285626de2b919b9f98f784c41763e5c2c914d not found: ID does not exist"
Sep 17 18:04:22.052270 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.052231    2580 scope.go:117] "RemoveContainer" containerID="e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab"
Sep 17 18:04:22.052473 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:04:22.052455    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab\": container with ID starting with e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab not found: ID does not exist" containerID="e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab"
Sep 17 18:04:22.052520 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.052478    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab"} err="failed to get container status \"e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab\": rpc error: code = NotFound desc = could not find container \"e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab\": container with ID starting with e8a4f5264b75fc168824f7818c69d9327896739b5538cf76b4f73e4158f2d2ab not found: ID does not exist"
Sep 17 18:04:22.057398 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.057370    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"]
Sep 17 18:04:22.060553 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:22.060532    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-69665bb668-9r92t"]
Sep 17 18:04:24.053539 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:24.053507    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" path="/var/lib/kubelet/pods/c227aac9-2c77-4501-9c1c-1dcd24bfd36a/volumes"
Sep 17 18:04:24.053974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:24.053946    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" path="/var/lib/kubelet/pods/f09baff7-f8aa-44dd-ba95-6263b3d56a9d/volumes"
Sep 17 18:04:46.932568 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:46.932526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:47.033080 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:47.033042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:53.029619 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.029585    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"]
Sep 17 18:04:53.029978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.029951    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerName="storage-initializer"
Sep 17 18:04:53.029978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.029963    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerName="storage-initializer"
Sep 17 18:04:53.029978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.029973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="main"
Sep 17 18:04:53.029978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.029978    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="main"
Sep 17 18:04:53.030130 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.029985    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerName="main"
Sep 17 18:04:53.030130 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.029990    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerName="main"
Sep 17 18:04:53.030130 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.030011    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="storage-initializer"
Sep 17 18:04:53.030130 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.030016    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="storage-initializer"
Sep 17 18:04:53.030130 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.030100    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f09baff7-f8aa-44dd-ba95-6263b3d56a9d" containerName="main"
Sep 17 18:04:53.030130 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.030110    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c227aac9-2c77-4501-9c1c-1dcd24bfd36a" containerName="main"
Sep 17 18:04:53.033328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.033307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.036866 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.036842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 18:04:53.037003 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.036865    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:04:53.037003 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.036880    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:04:53.037159 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.037119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-xsnlj\""
Sep 17 18:04:53.037159 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.037151    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:04:53.037280 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.037183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:04:53.052727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.052696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"]
Sep 17 18:04:53.063914 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.063879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:53.103975 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.103951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kserve-provision-location\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.104110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.103979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-model-cache\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.104110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.103996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tls-certs\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.104110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.104029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824bz\" (UniqueName: \"kubernetes.io/projected/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kube-api-access-824bz\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.104110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.104107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-home\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.104302 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.104136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40275-eafc-4e8d-84fe-5fd4de879b09-cabundle-cert\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.104302 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.104177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-dshm\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.104302 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.104198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tmp-dir\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-home\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40275-eafc-4e8d-84fe-5fd4de879b09-cabundle-cert\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-dshm\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204880 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tmp-dir\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204880 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kserve-provision-location\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204994 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-model-cache\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204994 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tls-certs\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.204994 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.204953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824bz\" (UniqueName: \"kubernetes.io/projected/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kube-api-access-824bz\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.205242 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.205221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tmp-dir\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.205312 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.205252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-model-cache\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.205312 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.205297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kserve-provision-location\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.205430 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.205417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-home\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.205551 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.205530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40275-eafc-4e8d-84fe-5fd4de879b09-cabundle-cert\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.207075 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.207059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-dshm\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.207436 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.207418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tls-certs\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.213569 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.213545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824bz\" (UniqueName: \"kubernetes.io/projected/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kube-api-access-824bz\") pod \"auth-disabled-test-kserve-5cd4c4c97d-9kbmg\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.314505 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.314434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:53.344173 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.344154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:04:53.635833 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:53.635802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"]
Sep 17 18:04:53.636451 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:04:53.636424    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd40275_eafc_4e8d_84fe_5fd4de879b09.slice/crio-4566021dfe39ef2cecbb77e56459ec80b9d9cddfc517a2c9d3ff9feb4df5aea9 WatchSource:0}: Error finding container 4566021dfe39ef2cecbb77e56459ec80b9d9cddfc517a2c9d3ff9feb4df5aea9: Status 404 returned error can't find the container with id 4566021dfe39ef2cecbb77e56459ec80b9d9cddfc517a2c9d3ff9feb4df5aea9
Sep 17 18:04:54.192237 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:54.192199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" event={"ID":"fdd40275-eafc-4e8d-84fe-5fd4de879b09","Type":"ContainerStarted","Data":"895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc"}
Sep 17 18:04:54.192621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:54.192256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" event={"ID":"fdd40275-eafc-4e8d-84fe-5fd4de879b09","Type":"ContainerStarted","Data":"4566021dfe39ef2cecbb77e56459ec80b9d9cddfc517a2c9d3ff9feb4df5aea9"}
Sep 17 18:04:55.693049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:55.693009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:56.204748 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:56.204720    2580 generic.go:356] "Generic (PLEG): container finished" podID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerID="895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc" exitCode=0
Sep 17 18:04:56.204870 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:56.204774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" event={"ID":"fdd40275-eafc-4e8d-84fe-5fd4de879b09","Type":"ContainerDied","Data":"895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc"}
Sep 17 18:04:56.205973 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:04:56.205957    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:05:02.226069 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:02.226018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:02.590681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:02.590591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:10.979827 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:10.979774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:11.151657 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:11.151564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:18.776694 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:18.776653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:24.409387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:24.409347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" event={"ID":"fdd40275-eafc-4e8d-84fe-5fd4de879b09","Type":"ContainerStarted","Data":"07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f"}
Sep 17 18:05:24.438547 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:24.438455    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podStartSLOduration=1.260966676 podStartE2EDuration="31.438438407s" totalImagesPullingTime="27.961598717s" totalInitContainerRuntime="2.215873014s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:24.435284752 +0000 UTC m=+1500.774678342" watchObservedRunningTime="2026-09-17 18:05:24.438438407 +0000 UTC m=+1500.777831998"
Sep 17 18:05:25.142343 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:25.142312    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:05:25.147712 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:25.147679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:05:25.344545 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:25.344505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:05:26.054774 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:26.054741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:26.153591 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:26.153562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:33.344594 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:33.344554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:05:33.346344 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:33.346312    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:05:39.028284 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.028243    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"]
Sep 17 18:05:39.110713 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.110674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:39.371066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.370987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:39.394831 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.394798    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"]
Sep 17 18:05:39.394984 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.394845    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"]
Sep 17 18:05:39.394984 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.394895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.397987 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.397965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:05:39.398162 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.398033    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:05:39.398252 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.398201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:05:39.399422 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.399371    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:05:39.399555 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.399425    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-46sls\""
Sep 17 18:05:39.399555 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.399436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:05:39.451397 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.451364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"]
Sep 17 18:05:39.451543 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.451452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.467211 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.467174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:39.512753 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.512727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-home\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.512905 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.512758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-dshm\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.512905 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.512787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tls-certs\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.512905 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.512809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.513101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.512906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqqx\" (UniqueName: \"kubernetes.io/projected/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kube-api-access-gzqqx\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.513101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.512969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.513101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.512994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-model-cache\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.513101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.513059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.614460 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.614460 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-home\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.614692 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.614692 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.614692 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-dshm\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.614692 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tls-certs\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.614692 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53102a44-7e40-479c-9565-7554e46b2987-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.614692 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.614692 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqqx\" (UniqueName: \"kubernetes.io/projected/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kube-api-access-gzqqx\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.615140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.615140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53102a44-7e40-479c-9565-7554e46b2987-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.615140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-model-cache\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.615140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.615009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.615140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.615031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.615140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.614799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-home\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.615507 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.615289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-model-cache\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.615572 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.615506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.616868 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.616848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-dshm\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.617287 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.617262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tls-certs\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.623939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.623887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqqx\" (UniqueName: \"kubernetes.io/projected/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kube-api-access-gzqqx\") pod \"tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.705608 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.705577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:39.715887 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.715860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53102a44-7e40-479c-9565-7554e46b2987-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.715985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.715911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.715985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.715932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.715985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.715968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53102a44-7e40-479c-9565-7554e46b2987-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.716323 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.716300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.716365 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.716327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.716596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.716579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53102a44-7e40-479c-9565-7554e46b2987-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.718295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.718275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53102a44-7e40-479c-9565-7554e46b2987-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:39.760859 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:39.760825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:40.024218 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:40.024187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"]
Sep 17 18:05:40.025113 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:05:40.025063    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode729ef9b_b24c_4b17_a168_f7aa081cb3b9.slice/crio-2ac833f411157a0178448897fd3d10d491cb4cd0bc7ffa75712cee6bc4bc8d3f WatchSource:0}: Error finding container 2ac833f411157a0178448897fd3d10d491cb4cd0bc7ffa75712cee6bc4bc8d3f: Status 404 returned error can't find the container with id 2ac833f411157a0178448897fd3d10d491cb4cd0bc7ffa75712cee6bc4bc8d3f
Sep 17 18:05:40.090398 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:40.090278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"]
Sep 17 18:05:40.092900 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:05:40.092876    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53102a44_7e40_479c_9565_7554e46b2987.slice/crio-d0e5e15dbdfae8800e064e23d50590458a49349e699d709ac0e4573f88788227 WatchSource:0}: Error finding container d0e5e15dbdfae8800e064e23d50590458a49349e699d709ac0e4573f88788227: Status 404 returned error can't find the container with id d0e5e15dbdfae8800e064e23d50590458a49349e699d709ac0e4573f88788227
Sep 17 18:05:40.509788 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:40.509750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" event={"ID":"53102a44-7e40-479c-9565-7554e46b2987","Type":"ContainerStarted","Data":"711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84"}
Sep 17 18:05:40.509963 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:40.509836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" event={"ID":"53102a44-7e40-479c-9565-7554e46b2987","Type":"ContainerStarted","Data":"d0e5e15dbdfae8800e064e23d50590458a49349e699d709ac0e4573f88788227"}
Sep 17 18:05:40.511737 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:40.511704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" event={"ID":"e729ef9b-b24c-4b17-a168-f7aa081cb3b9","Type":"ContainerStarted","Data":"ef27f6021f5dcef10213f27c4bce189de924eaa5b5e80d9d6740bc0be587aabb"}
Sep 17 18:05:40.511886 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:40.511756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" event={"ID":"e729ef9b-b24c-4b17-a168-f7aa081cb3b9","Type":"ContainerStarted","Data":"2ac833f411157a0178448897fd3d10d491cb4cd0bc7ffa75712cee6bc4bc8d3f"}
Sep 17 18:05:43.345646 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:43.345606    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:05:43.530717 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:43.530674    2580 generic.go:356] "Generic (PLEG): container finished" podID="53102a44-7e40-479c-9565-7554e46b2987" containerID="711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84" exitCode=0
Sep 17 18:05:43.530900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:43.530754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" event={"ID":"53102a44-7e40-479c-9565-7554e46b2987","Type":"ContainerDied","Data":"711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84"}
Sep 17 18:05:43.532565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:43.532543    2580 generic.go:356] "Generic (PLEG): container finished" podID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerID="ef27f6021f5dcef10213f27c4bce189de924eaa5b5e80d9d6740bc0be587aabb" exitCode=0
Sep 17 18:05:43.532707 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:43.532583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" event={"ID":"e729ef9b-b24c-4b17-a168-f7aa081cb3b9","Type":"ContainerDied","Data":"ef27f6021f5dcef10213f27c4bce189de924eaa5b5e80d9d6740bc0be587aabb"}
Sep 17 18:05:44.541733 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:44.541645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" event={"ID":"53102a44-7e40-479c-9565-7554e46b2987","Type":"ContainerStarted","Data":"6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781"}
Sep 17 18:05:44.542227 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:44.541890    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:05:44.543976 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:44.543953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" event={"ID":"e729ef9b-b24c-4b17-a168-f7aa081cb3b9","Type":"ContainerStarted","Data":"d3cf2121d9977ba574472c7a303e5bcb24bda8f9c5446f6042dcfb12725b671b"}
Sep 17 18:05:44.564539 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:44.564489    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" podStartSLOduration=2.98805654 podStartE2EDuration="5.564473968s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.576417428s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:44.563320781 +0000 UTC m=+1520.902714372" watchObservedRunningTime="2026-09-17 18:05:44.564473968 +0000 UTC m=+1520.903867560"
Sep 17 18:05:44.591267 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:44.591225    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" podStartSLOduration=3.054709411 podStartE2EDuration="5.591212429s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.536503018s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:44.588515592 +0000 UTC m=+1520.927909188" watchObservedRunningTime="2026-09-17 18:05:44.591212429 +0000 UTC m=+1520.930606020"
Sep 17 18:05:44.706293 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:44.706264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:49.706168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:49.706133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:49.719117 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:49.719068    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:50.594697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:50.594664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:05:51.711401 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:51.711337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:53.345565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:05:53.345523    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:06:03.344869 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:03.344824    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:06:13.344946 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:13.344902    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:06:15.562096 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:15.562050    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:06:16.520342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:16.520302    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8_53102a44-7e40-479c-9565-7554e46b2987/main/0.log"
Sep 17 18:06:17.446179 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.446142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:17.447898 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.447869    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"]
Sep 17 18:06:17.457369 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:06:17.457338    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:06:17.457369 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:17.457369 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:17.457369 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="d3cf2121d9977ba574472c7a303e5bcb24bda8f9c5446f6042dcfb12725b671b" cmd=["/bin/sleep","15"]
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.457386    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" message="" err=<
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:06:17.457398    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="main" containerID="cri-o://d3cf2121d9977ba574472c7a303e5bcb24bda8f9c5446f6042dcfb12725b671b"
Sep 17 18:06:17.457596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.457440    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="main" containerID="cri-o://d3cf2121d9977ba574472c7a303e5bcb24bda8f9c5446f6042dcfb12725b671b" gracePeriod=60
Sep 17 18:06:17.468622 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.468599    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"]
Sep 17 18:06:17.468807 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.468789    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" podUID="53102a44-7e40-479c-9565-7554e46b2987" containerName="main" containerID="cri-o://6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781" gracePeriod=30
Sep 17 18:06:17.527907 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.527873    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:17.584046 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.584000    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 17 18:06:17.746458 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.746371    2580 generic.go:356] "Generic (PLEG): container finished" podID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerID="d3cf2121d9977ba574472c7a303e5bcb24bda8f9c5446f6042dcfb12725b671b" exitCode=0
Sep 17 18:06:17.746616 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:17.746487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" event={"ID":"e729ef9b-b24c-4b17-a168-f7aa081cb3b9","Type":"ContainerDied","Data":"d3cf2121d9977ba574472c7a303e5bcb24bda8f9c5446f6042dcfb12725b671b"}
Sep 17 18:06:18.138682 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.138654    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:06:18.208476 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.208453    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:06:18.272801 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.272773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-tmp-dir\") pod \"53102a44-7e40-479c-9565-7554e46b2987\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") "
Sep 17 18:06:18.272942 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.272814    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-kserve-provision-location\") pod \"53102a44-7e40-479c-9565-7554e46b2987\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") "
Sep 17 18:06:18.272942 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.272885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53102a44-7e40-479c-9565-7554e46b2987-tls-certs\") pod \"53102a44-7e40-479c-9565-7554e46b2987\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") "
Sep 17 18:06:18.272942 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.272905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53102a44-7e40-479c-9565-7554e46b2987-cabundle-cert\") pod \"53102a44-7e40-479c-9565-7554e46b2987\" (UID: \"53102a44-7e40-479c-9565-7554e46b2987\") "
Sep 17 18:06:18.273332 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.273297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53102a44-7e40-479c-9565-7554e46b2987" (UID: "53102a44-7e40-479c-9565-7554e46b2987"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.273461 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.273310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53102a44-7e40-479c-9565-7554e46b2987-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53102a44-7e40-479c-9565-7554e46b2987" (UID: "53102a44-7e40-479c-9565-7554e46b2987"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:18.275095 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.275059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53102a44-7e40-479c-9565-7554e46b2987-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53102a44-7e40-479c-9565-7554e46b2987" (UID: "53102a44-7e40-479c-9565-7554e46b2987"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:18.286439 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.286416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53102a44-7e40-479c-9565-7554e46b2987" (UID: "53102a44-7e40-479c-9565-7554e46b2987"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.373635 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.373567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzqqx\" (UniqueName: \"kubernetes.io/projected/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kube-api-access-gzqqx\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.373635 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.373608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-model-cache\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.373832 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.373654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-cabundle-cert\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.373832 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.373714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kserve-provision-location\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.373832 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.373746    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-dshm\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.373832 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.373778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tmp-dir\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.374113 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.373914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.374113 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.374113 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tls-certs\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.374113 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:18.374322 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-home\") pod \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\" (UID: \"e729ef9b-b24c-4b17-a168-f7aa081cb3b9\") "
Sep 17 18:06:18.374380 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-home" (OuterVolumeSpecName: "home") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.374436 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374379    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.374436 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374396    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.374436 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374411    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53102a44-7e40-479c-9565-7554e46b2987-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.374436 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374425    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.374598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374440    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.374598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374453    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53102a44-7e40-479c-9565-7554e46b2987-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.374598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.374466    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53102a44-7e40-479c-9565-7554e46b2987-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.376137 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.376112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-dshm" (OuterVolumeSpecName: "dshm") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.376308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.376287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kube-api-access-gzqqx" (OuterVolumeSpecName: "kube-api-access-gzqqx") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "kube-api-access-gzqqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:18.376602 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.376566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:18.389189 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.389166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e729ef9b-b24c-4b17-a168-f7aa081cb3b9" (UID: "e729ef9b-b24c-4b17-a168-f7aa081cb3b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.475618 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.475594    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.475618 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.475616    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.475955 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.475625    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.475955 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.475635    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.475955 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.475644    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqqx\" (UniqueName: \"kubernetes.io/projected/e729ef9b-b24c-4b17-a168-f7aa081cb3b9-kube-api-access-gzqqx\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.753831 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.753791    2580 generic.go:356] "Generic (PLEG): container finished" podID="53102a44-7e40-479c-9565-7554e46b2987" containerID="6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781" exitCode=0
Sep 17 18:06:18.754020 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.753859    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"
Sep 17 18:06:18.754020 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.753860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" event={"ID":"53102a44-7e40-479c-9565-7554e46b2987","Type":"ContainerDied","Data":"6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781"}
Sep 17 18:06:18.754020 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.753973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8" event={"ID":"53102a44-7e40-479c-9565-7554e46b2987","Type":"ContainerDied","Data":"d0e5e15dbdfae8800e064e23d50590458a49349e699d709ac0e4573f88788227"}
Sep 17 18:06:18.754020 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.754005    2580 scope.go:117] "RemoveContainer" containerID="6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781"
Sep 17 18:06:18.756438 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.756416    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"
Sep 17 18:06:18.756555 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.756417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh" event={"ID":"e729ef9b-b24c-4b17-a168-f7aa081cb3b9","Type":"ContainerDied","Data":"2ac833f411157a0178448897fd3d10d491cb4cd0bc7ffa75712cee6bc4bc8d3f"}
Sep 17 18:06:18.765892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.765872    2580 scope.go:117] "RemoveContainer" containerID="711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84"
Sep 17 18:06:18.775724 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.775700    2580 scope.go:117] "RemoveContainer" containerID="6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781"
Sep 17 18:06:18.776064 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:06:18.776008    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781\": container with ID starting with 6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781 not found: ID does not exist" containerID="6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781"
Sep 17 18:06:18.776175 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.776071    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781"} err="failed to get container status \"6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781\": rpc error: code = NotFound desc = could not find container \"6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781\": container with ID starting with 6e0f06ab9d1566c03e1796c933d98cf7483abf0cb92b58e931475ef904d91781 not found: ID does not exist"
Sep 17 18:06:18.776175 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.776102    2580 scope.go:117] "RemoveContainer" containerID="711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84"
Sep 17 18:06:18.776675 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:06:18.776554    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84\": container with ID starting with 711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84 not found: ID does not exist" containerID="711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84"
Sep 17 18:06:18.776675 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.776586    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84"} err="failed to get container status \"711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84\": rpc error: code = NotFound desc = could not find container \"711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84\": container with ID starting with 711fe829d8372e416ed395cbb8367938459486eade1e7010a1f40eed4197dc84 not found: ID does not exist"
Sep 17 18:06:18.776675 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.776605    2580 scope.go:117] "RemoveContainer" containerID="d3cf2121d9977ba574472c7a303e5bcb24bda8f9c5446f6042dcfb12725b671b"
Sep 17 18:06:18.777049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.777001    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"]
Sep 17 18:06:18.779573 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.779552    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7856d48f66-7dxk8"]
Sep 17 18:06:18.786241 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.786222    2580 scope.go:117] "RemoveContainer" containerID="ef27f6021f5dcef10213f27c4bce189de924eaa5b5e80d9d6740bc0be587aabb"
Sep 17 18:06:18.791078 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.791056    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"]
Sep 17 18:06:18.794467 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:18.794448    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54fb5fc6d8-cfsgh"]
Sep 17 18:06:20.047686 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:20.047647    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53102a44-7e40-479c-9565-7554e46b2987" path="/var/lib/kubelet/pods/53102a44-7e40-479c-9565-7554e46b2987/volumes"
Sep 17 18:06:20.048315 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:20.048294    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" path="/var/lib/kubelet/pods/e729ef9b-b24c-4b17-a168-f7aa081cb3b9/volumes"
Sep 17 18:06:23.344955 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:23.344915    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:06:33.345265 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:33.345216    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:06:43.344896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:43.344849    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:06:53.345431 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:06:53.345389    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:07:03.344643 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:03.344599    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:07:06.939669 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:06.939626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:08.224894 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:08.224851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:13.354214 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:13.354187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:07:13.361639 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:13.361619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:07:14.770740 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:14.770660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:16.825675 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.825644    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"]
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53102a44-7e40-479c-9565-7554e46b2987" containerName="storage-initializer"
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826032    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53102a44-7e40-479c-9565-7554e46b2987" containerName="storage-initializer"
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826044    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="main"
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826050    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="main"
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826058    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53102a44-7e40-479c-9565-7554e46b2987" containerName="main"
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53102a44-7e40-479c-9565-7554e46b2987" containerName="main"
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826076    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="storage-initializer"
Sep 17 18:07:16.826074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826095    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="storage-initializer"
Sep 17 18:07:16.826359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826156    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e729ef9b-b24c-4b17-a168-f7aa081cb3b9" containerName="main"
Sep 17 18:07:16.826359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.826163    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53102a44-7e40-479c-9565-7554e46b2987" containerName="main"
Sep 17 18:07:16.832208 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.832188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:16.836391 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.836360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-b9848\""
Sep 17 18:07:16.836511 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.836419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 18:07:16.836511 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.836419    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 18:07:16.836511 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.836504    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 18:07:16.843013 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.842991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"]
Sep 17 18:07:16.851552 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.851528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:16.945584 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.945562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:16.945680 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.945615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:16.945680 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.945659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h66q\" (UniqueName: \"kubernetes.io/projected/3839eb00-5e48-4a17-ba35-31c2f7187264-kube-api-access-9h66q\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:16.945773 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.945685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3839eb00-5e48-4a17-ba35-31c2f7187264-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:16.945773 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.945728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:16.945773 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:16.945748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.046439 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h66q\" (UniqueName: \"kubernetes.io/projected/3839eb00-5e48-4a17-ba35-31c2f7187264-kube-api-access-9h66q\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.046555 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3839eb00-5e48-4a17-ba35-31c2f7187264-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.046555 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.046555 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.046716 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.046716 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.046945 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.047052 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.046962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.047052 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.047027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.049260 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.049239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.049705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.049687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3839eb00-5e48-4a17-ba35-31c2f7187264-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.070970 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.070944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h66q\" (UniqueName: \"kubernetes.io/projected/3839eb00-5e48-4a17-ba35-31c2f7187264-kube-api-access-9h66q\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.144209 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.144189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:17.148407 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.148115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:17.455668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:17.455646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"]
Sep 17 18:07:17.458233 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:07:17.458207    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3839eb00_5e48_4a17_ba35_31c2f7187264.slice/crio-d383beb15eef03b180f5606a93240513fbd0f0f41d7135a5fc64d6dd5a9c0d86 WatchSource:0}: Error finding container d383beb15eef03b180f5606a93240513fbd0f0f41d7135a5fc64d6dd5a9c0d86: Status 404 returned error can't find the container with id d383beb15eef03b180f5606a93240513fbd0f0f41d7135a5fc64d6dd5a9c0d86
Sep 17 18:07:18.101607 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:18.101573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" event={"ID":"3839eb00-5e48-4a17-ba35-31c2f7187264","Type":"ContainerStarted","Data":"f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d"}
Sep 17 18:07:18.101967 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:18.101621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" event={"ID":"3839eb00-5e48-4a17-ba35-31c2f7187264","Type":"ContainerStarted","Data":"d383beb15eef03b180f5606a93240513fbd0f0f41d7135a5fc64d6dd5a9c0d86"}
Sep 17 18:07:18.145124 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:18.145072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:21.554036 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:21.554001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:21.635072 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:21.635037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:27.144740 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:27.144709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:27.158118 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:27.158074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:27.180023 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:27.179982    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" podStartSLOduration=11.179969373 podStartE2EDuration="11.179969373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:18.12189999 +0000 UTC m=+1614.461293582" watchObservedRunningTime="2026-09-17 18:07:27.179969373 +0000 UTC m=+1623.519362965"
Sep 17 18:07:28.156763 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:28.156736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:37.368306 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:37.368272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:43.117359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:43.117325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:43.184918 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:43.184886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:43.972682 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:43.972649    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"]
Sep 17 18:07:44.038376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:44.038340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:44.137387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:44.137337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:50.252105 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.252054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:50.259617 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.259590    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"]
Sep 17 18:07:50.268725 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:07:50.268696    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:07:50.268725 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:07:50.268725 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:07:50.268725 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d" cmd=["/bin/sleep","15"]
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.268741    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" message="" err=<
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:07:50.268752    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" podUID="3839eb00-5e48-4a17-ba35-31c2f7187264" containerName="main" containerID="cri-o://f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d"
Sep 17 18:07:50.269041 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.268787    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" podUID="3839eb00-5e48-4a17-ba35-31c2f7187264" containerName="main" containerID="cri-o://f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d" gracePeriod=60
Sep 17 18:07:50.330180 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.330146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:50.616544 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.616523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:50.705285 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-dshm\") pod \"3839eb00-5e48-4a17-ba35-31c2f7187264\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") "
Sep 17 18:07:50.705421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-home\") pod \"3839eb00-5e48-4a17-ba35-31c2f7187264\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") "
Sep 17 18:07:50.705421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h66q\" (UniqueName: \"kubernetes.io/projected/3839eb00-5e48-4a17-ba35-31c2f7187264-kube-api-access-9h66q\") pod \"3839eb00-5e48-4a17-ba35-31c2f7187264\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") "
Sep 17 18:07:50.705421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-tmp-dir\") pod \"3839eb00-5e48-4a17-ba35-31c2f7187264\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") "
Sep 17 18:07:50.705545 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-model-cache\") pod \"3839eb00-5e48-4a17-ba35-31c2f7187264\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") "
Sep 17 18:07:50.705545 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3839eb00-5e48-4a17-ba35-31c2f7187264-tls-certs\") pod \"3839eb00-5e48-4a17-ba35-31c2f7187264\" (UID: \"3839eb00-5e48-4a17-ba35-31c2f7187264\") "
Sep 17 18:07:50.705652 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-home" (OuterVolumeSpecName: "home") pod "3839eb00-5e48-4a17-ba35-31c2f7187264" (UID: "3839eb00-5e48-4a17-ba35-31c2f7187264"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:50.705652 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3839eb00-5e48-4a17-ba35-31c2f7187264" (UID: "3839eb00-5e48-4a17-ba35-31c2f7187264"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:50.705764 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-model-cache" (OuterVolumeSpecName: "model-cache") pod "3839eb00-5e48-4a17-ba35-31c2f7187264" (UID: "3839eb00-5e48-4a17-ba35-31c2f7187264"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:50.705823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705789    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:50.705823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705801    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:50.705823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.705810    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:50.707560 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.707533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3839eb00-5e48-4a17-ba35-31c2f7187264-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3839eb00-5e48-4a17-ba35-31c2f7187264" (UID: "3839eb00-5e48-4a17-ba35-31c2f7187264"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:50.707663 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.707572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-dshm" (OuterVolumeSpecName: "dshm") pod "3839eb00-5e48-4a17-ba35-31c2f7187264" (UID: "3839eb00-5e48-4a17-ba35-31c2f7187264"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:50.707663 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.707585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3839eb00-5e48-4a17-ba35-31c2f7187264-kube-api-access-9h66q" (OuterVolumeSpecName: "kube-api-access-9h66q") pod "3839eb00-5e48-4a17-ba35-31c2f7187264" (UID: "3839eb00-5e48-4a17-ba35-31c2f7187264"). InnerVolumeSpecName "kube-api-access-9h66q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:50.806621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.806570    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3839eb00-5e48-4a17-ba35-31c2f7187264-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:50.806621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.806591    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3839eb00-5e48-4a17-ba35-31c2f7187264-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:50.806621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:50.806599    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h66q\" (UniqueName: \"kubernetes.io/projected/3839eb00-5e48-4a17-ba35-31c2f7187264-kube-api-access-9h66q\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:51.284807 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.284772    2580 generic.go:356] "Generic (PLEG): container finished" podID="3839eb00-5e48-4a17-ba35-31c2f7187264" containerID="f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d" exitCode=0
Sep 17 18:07:51.285233 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.284817    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"
Sep 17 18:07:51.285233 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.284822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" event={"ID":"3839eb00-5e48-4a17-ba35-31c2f7187264","Type":"ContainerDied","Data":"f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d"}
Sep 17 18:07:51.285233 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.284875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2" event={"ID":"3839eb00-5e48-4a17-ba35-31c2f7187264","Type":"ContainerDied","Data":"d383beb15eef03b180f5606a93240513fbd0f0f41d7135a5fc64d6dd5a9c0d86"}
Sep 17 18:07:51.285233 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.284905    2580 scope.go:117] "RemoveContainer" containerID="f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d"
Sep 17 18:07:51.294803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.294778    2580 scope.go:117] "RemoveContainer" containerID="f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d"
Sep 17 18:07:51.295054 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:07:51.295034    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d\": container with ID starting with f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d not found: ID does not exist" containerID="f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d"
Sep 17 18:07:51.295156 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.295064    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d"} err="failed to get container status \"f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d\": rpc error: code = NotFound desc = could not find container \"f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d\": container with ID starting with f2dab2ca5e8e0c94b3399f610e83bc1c40efe7078169e3531c3fae6ce669699d not found: ID does not exist"
Sep 17 18:07:51.308356 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.308334    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"]
Sep 17 18:07:51.312482 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:51.312462    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-x6tc2"]
Sep 17 18:07:52.046498 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:52.046462    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3839eb00-5e48-4a17-ba35-31c2f7187264" path="/var/lib/kubelet/pods/3839eb00-5e48-4a17-ba35-31c2f7187264/volumes"
Sep 17 18:07:58.996575 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:58.996524    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" containerID="cri-o://07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f" gracePeriod=45
Sep 17 18:07:59.355743 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.355655    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 18:07:59.745934 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.745912    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:07:59.877892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.877832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40275-eafc-4e8d-84fe-5fd4de879b09-cabundle-cert\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.877892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.877883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tls-certs\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.878049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.877915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kserve-provision-location\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.878049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.877944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tmp-dir\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.878049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.877977    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-model-cache\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.878049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.878017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-dshm\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.878273 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.878048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-824bz\" (UniqueName: \"kubernetes.io/projected/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kube-api-access-824bz\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.878273 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.878074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-home\") pod \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\" (UID: \"fdd40275-eafc-4e8d-84fe-5fd4de879b09\") "
Sep 17 18:07:59.878383 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.878319    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:59.878383 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.878256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd40275-eafc-4e8d-84fe-5fd4de879b09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:59.878974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.878948    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-home" (OuterVolumeSpecName: "home") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:59.880445 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.880415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kube-api-access-824bz" (OuterVolumeSpecName: "kube-api-access-824bz") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "kube-api-access-824bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:59.880813 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.880785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-dshm" (OuterVolumeSpecName: "dshm") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:59.880907 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.880805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:59.886145 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.886120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:59.891494 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.891471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdd40275-eafc-4e8d-84fe-5fd4de879b09" (UID: "fdd40275-eafc-4e8d-84fe-5fd4de879b09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:59.978693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978666    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:59.978693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978692    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:59.978817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978701    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:59.978817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978709    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:59.978817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978718    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:59.978817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978726    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824bz\" (UniqueName: \"kubernetes.io/projected/fdd40275-eafc-4e8d-84fe-5fd4de879b09-kube-api-access-824bz\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:59.978817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978735    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd40275-eafc-4e8d-84fe-5fd4de879b09-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:07:59.978817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:07:59.978743    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40275-eafc-4e8d-84fe-5fd4de879b09-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:08:00.338010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.337976    2580 generic.go:356] "Generic (PLEG): container finished" podID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerID="07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f" exitCode=0
Sep 17 18:08:00.338504 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.338043    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"
Sep 17 18:08:00.338504 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.338063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" event={"ID":"fdd40275-eafc-4e8d-84fe-5fd4de879b09","Type":"ContainerDied","Data":"07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f"}
Sep 17 18:08:00.338504 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.338163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg" event={"ID":"fdd40275-eafc-4e8d-84fe-5fd4de879b09","Type":"ContainerDied","Data":"4566021dfe39ef2cecbb77e56459ec80b9d9cddfc517a2c9d3ff9feb4df5aea9"}
Sep 17 18:08:00.338504 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.338200    2580 scope.go:117] "RemoveContainer" containerID="07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f"
Sep 17 18:08:00.348535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.348508    2580 scope.go:117] "RemoveContainer" containerID="895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc"
Sep 17 18:08:00.357193 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.357170    2580 scope.go:117] "RemoveContainer" containerID="07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f"
Sep 17 18:08:00.357515 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:08:00.357491    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f\": container with ID starting with 07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f not found: ID does not exist" containerID="07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f"
Sep 17 18:08:00.357587 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.357523    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f"} err="failed to get container status \"07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f\": rpc error: code = NotFound desc = could not find container \"07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f\": container with ID starting with 07e3cfbe8d15f7d1222a21239c012d3c682831d2ba7c792eec57284037a2b27f not found: ID does not exist"
Sep 17 18:08:00.357587 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.357541    2580 scope.go:117] "RemoveContainer" containerID="895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc"
Sep 17 18:08:00.357903 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:08:00.357876    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc\": container with ID starting with 895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc not found: ID does not exist" containerID="895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc"
Sep 17 18:08:00.358018 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.357909    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc"} err="failed to get container status \"895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc\": rpc error: code = NotFound desc = could not find container \"895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc\": container with ID starting with 895678c1d1f46011385abec41a99affeb7c5fee0c119cbf5eaf608dd3e66babc not found: ID does not exist"
Sep 17 18:08:00.360162 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.360137    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"]
Sep 17 18:08:00.366714 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:00.366690    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cd4c4c97d-9kbmg"]
Sep 17 18:08:02.046947 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:02.046864    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" path="/var/lib/kubelet/pods/fdd40275-eafc-4e8d-84fe-5fd4de879b09/volumes"
Sep 17 18:08:33.960523 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:33.960474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:34.235969 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:34.235885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:34.447400 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:34.447368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:37.556341 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:37.556305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.304037 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:40.303996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.405470 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:40.405443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:41.094931 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:41.094896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:44.290754 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.290720    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"]
Sep 17 18:08:44.291148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291075    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main"
Sep 17 18:08:44.291148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291103    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main"
Sep 17 18:08:44.291148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3839eb00-5e48-4a17-ba35-31c2f7187264" containerName="main"
Sep 17 18:08:44.291148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291124    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3839eb00-5e48-4a17-ba35-31c2f7187264" containerName="main"
Sep 17 18:08:44.291148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291142    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="storage-initializer"
Sep 17 18:08:44.291148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291149    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="storage-initializer"
Sep 17 18:08:44.291367 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd40275-eafc-4e8d-84fe-5fd4de879b09" containerName="main"
Sep 17 18:08:44.291367 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.291215    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3839eb00-5e48-4a17-ba35-31c2f7187264" containerName="main"
Sep 17 18:08:44.296742 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.296667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.301143 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.301117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:08:44.301291 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.301145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 18:08:44.301291 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.301165    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-984hk\""
Sep 17 18:08:44.301291 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.301235    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:08:44.307993 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.307969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"]
Sep 17 18:08:44.321310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.321289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:44.394895 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.394868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.395010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.394904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.395010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.394927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7164d67-877d-41e6-b0d2-6acfc675711d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.395010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.394956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.395010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.394984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.395198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.395024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrnwb\" (UniqueName: \"kubernetes.io/projected/f7164d67-877d-41e6-b0d2-6acfc675711d-kube-api-access-xrnwb\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7164d67-877d-41e6-b0d2-6acfc675711d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrnwb\" (UniqueName: \"kubernetes.io/projected/f7164d67-877d-41e6-b0d2-6acfc675711d-kube-api-access-xrnwb\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496481 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496588 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496714 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.496714 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.496702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.498476 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.498457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.498837 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.498820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7164d67-877d-41e6-b0d2-6acfc675711d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.505597 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.505578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrnwb\" (UniqueName: \"kubernetes.io/projected/f7164d67-877d-41e6-b0d2-6acfc675711d-kube-api-access-xrnwb\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.609248 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.609197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:44.897434 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:44.897403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"]
Sep 17 18:08:44.898747 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:08:44.898723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7164d67_877d_41e6_b0d2_6acfc675711d.slice/crio-476ab85308a34198aa7fc7ece68d3d2d469578d7307861847743fc52db4d62b0 WatchSource:0}: Error finding container 476ab85308a34198aa7fc7ece68d3d2d469578d7307861847743fc52db4d62b0: Status 404 returned error can't find the container with id 476ab85308a34198aa7fc7ece68d3d2d469578d7307861847743fc52db4d62b0
Sep 17 18:08:45.594039 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:45.594008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" event={"ID":"f7164d67-877d-41e6-b0d2-6acfc675711d","Type":"ContainerStarted","Data":"fa06828f74d2a913ac5c3dbb0382638833485de3e3d4a8f30d604d769a3c46bf"}
Sep 17 18:08:45.594421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:45.594059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" event={"ID":"f7164d67-877d-41e6-b0d2-6acfc675711d","Type":"ContainerStarted","Data":"476ab85308a34198aa7fc7ece68d3d2d469578d7307861847743fc52db4d62b0"}
Sep 17 18:08:45.609822 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:45.609798    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:45.616795 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:45.616756    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" podStartSLOduration=1.616741252 podStartE2EDuration="1.616741252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:45.616524512 +0000 UTC m=+1701.955918104" watchObservedRunningTime="2026-09-17 18:08:45.616741252 +0000 UTC m=+1701.956134842"
Sep 17 18:08:47.587768 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:47.587730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:47.689978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:47.689940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:47.789247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:47.789213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:54.161192 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:54.161160    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:54.265785 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:54.265751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:54.609590 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:54.609527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:54.622425 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:54.622403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:08:54.653971 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:08:54.653950    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:09:05.856504 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:05.856471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:06.102273 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:06.102243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:14.244350 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:14.244312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:17.379214 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:17.379172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:18.587673 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:18.587630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:45.294191 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.294154    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"]
Sep 17 18:09:45.297371 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.297355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.301190 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.301164    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 18:09:45.301351 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.301209    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-zqr5c\""
Sep 17 18:09:45.301351 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.301237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:09:45.301351 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.301212    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:09:45.309383 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.309360    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"]
Sep 17 18:09:45.343157 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.343135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.343263 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.343180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.343263 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.343228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.343362 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.343278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7x9h\" (UniqueName: \"kubernetes.io/projected/426d3a06-5eb6-4daf-a65d-effa9f32132e-kube-api-access-q7x9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.343362 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.343308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.343362 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.343322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426d3a06-5eb6-4daf-a65d-effa9f32132e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.347396 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.347366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:45.444386 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.444358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.444483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.444393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.444483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.444420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7x9h\" (UniqueName: \"kubernetes.io/projected/426d3a06-5eb6-4daf-a65d-effa9f32132e-kube-api-access-q7x9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.444483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.444462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.444618 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.444485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426d3a06-5eb6-4daf-a65d-effa9f32132e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.444707 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.444687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.444850 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.444825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.445043 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.445024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.445152 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.445130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.446700 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.446681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.447227 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.447204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426d3a06-5eb6-4daf-a65d-effa9f32132e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.452555 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.452538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7x9h\" (UniqueName: \"kubernetes.io/projected/426d3a06-5eb6-4daf-a65d-effa9f32132e-kube-api-access-q7x9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.539410 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.539376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:45.608638 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.608572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:45.888147 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.887899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"]
Sep 17 18:09:45.890503 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:09:45.890475    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d3a06_5eb6_4daf_a65d_effa9f32132e.slice/crio-2e4549bceb5b6b89878752387462fa9980015b8d23de35ff4027240f7fe1bf6e WatchSource:0}: Error finding container 2e4549bceb5b6b89878752387462fa9980015b8d23de35ff4027240f7fe1bf6e: Status 404 returned error can't find the container with id 2e4549bceb5b6b89878752387462fa9980015b8d23de35ff4027240f7fe1bf6e
Sep 17 18:09:45.931067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:45.931002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" event={"ID":"426d3a06-5eb6-4daf-a65d-effa9f32132e","Type":"ContainerStarted","Data":"2e4549bceb5b6b89878752387462fa9980015b8d23de35ff4027240f7fe1bf6e"}
Sep 17 18:09:46.937817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:46.937780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" event={"ID":"426d3a06-5eb6-4daf-a65d-effa9f32132e","Type":"ContainerStarted","Data":"038ecefbf9e43f54b06e7cd78ba73426d3f1502469b71c5d859a3ed20d14afe1"}
Sep 17 18:09:46.957602 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:46.957552    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" podStartSLOduration=1.957536409 podStartE2EDuration="1.957536409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:46.956005279 +0000 UTC m=+1763.295398881" watchObservedRunningTime="2026-09-17 18:09:46.957536409 +0000 UTC m=+1763.296930043"
Sep 17 18:09:47.609719 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:47.609686    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:55.608955 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:55.608918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:55.621426 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:55.621401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:55.997105 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:55.997067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:09:56.672791 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:09:56.672741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:00.814542 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:00.814511    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"]
Sep 17 18:10:00.823602 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:10:00.823559    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:10:00.823602 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:00.823602 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:00.823602 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="fa06828f74d2a913ac5c3dbb0382638833485de3e3d4a8f30d604d769a3c46bf" cmd=["/bin/sleep","15"]
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:00.823625    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" message="" err=<
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:10:00.823645    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" podUID="f7164d67-877d-41e6-b0d2-6acfc675711d" containerName="main" containerID="cri-o://fa06828f74d2a913ac5c3dbb0382638833485de3e3d4a8f30d604d769a3c46bf"
Sep 17 18:10:00.823904 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:00.823697    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" podUID="f7164d67-877d-41e6-b0d2-6acfc675711d" containerName="main" containerID="cri-o://fa06828f74d2a913ac5c3dbb0382638833485de3e3d4a8f30d604d769a3c46bf" gracePeriod=60
Sep 17 18:10:00.838373 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:00.838329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:01.018579 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.018539    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7164d67-877d-41e6-b0d2-6acfc675711d" containerID="fa06828f74d2a913ac5c3dbb0382638833485de3e3d4a8f30d604d769a3c46bf" exitCode=0
Sep 17 18:10:01.018740 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.018603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" event={"ID":"f7164d67-877d-41e6-b0d2-6acfc675711d","Type":"ContainerDied","Data":"fa06828f74d2a913ac5c3dbb0382638833485de3e3d4a8f30d604d769a3c46bf"}
Sep 17 18:10:01.067340 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.067288    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:10:01.175930 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.175906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-tmp-dir\") pod \"f7164d67-877d-41e6-b0d2-6acfc675711d\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") "
Sep 17 18:10:01.176076 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.175949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7164d67-877d-41e6-b0d2-6acfc675711d-tls-certs\") pod \"f7164d67-877d-41e6-b0d2-6acfc675711d\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") "
Sep 17 18:10:01.176076 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.175972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-dshm\") pod \"f7164d67-877d-41e6-b0d2-6acfc675711d\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") "
Sep 17 18:10:01.176076 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.175991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-home\") pod \"f7164d67-877d-41e6-b0d2-6acfc675711d\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") "
Sep 17 18:10:01.176076 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.176031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrnwb\" (UniqueName: \"kubernetes.io/projected/f7164d67-877d-41e6-b0d2-6acfc675711d-kube-api-access-xrnwb\") pod \"f7164d67-877d-41e6-b0d2-6acfc675711d\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") "
Sep 17 18:10:01.176076 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.176064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-model-cache\") pod \"f7164d67-877d-41e6-b0d2-6acfc675711d\" (UID: \"f7164d67-877d-41e6-b0d2-6acfc675711d\") "
Sep 17 18:10:01.176385 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.176253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7164d67-877d-41e6-b0d2-6acfc675711d" (UID: "f7164d67-877d-41e6-b0d2-6acfc675711d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:01.176385 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.176310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-home" (OuterVolumeSpecName: "home") pod "f7164d67-877d-41e6-b0d2-6acfc675711d" (UID: "f7164d67-877d-41e6-b0d2-6acfc675711d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:01.176492 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.176438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7164d67-877d-41e6-b0d2-6acfc675711d" (UID: "f7164d67-877d-41e6-b0d2-6acfc675711d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:01.176492 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.176464    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:01.176492 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.176484    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:01.178505 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.178480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7164d67-877d-41e6-b0d2-6acfc675711d-kube-api-access-xrnwb" (OuterVolumeSpecName: "kube-api-access-xrnwb") pod "f7164d67-877d-41e6-b0d2-6acfc675711d" (UID: "f7164d67-877d-41e6-b0d2-6acfc675711d"). InnerVolumeSpecName "kube-api-access-xrnwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:01.178623 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.178612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-dshm" (OuterVolumeSpecName: "dshm") pod "f7164d67-877d-41e6-b0d2-6acfc675711d" (UID: "f7164d67-877d-41e6-b0d2-6acfc675711d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:01.178691 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.178624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7164d67-877d-41e6-b0d2-6acfc675711d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7164d67-877d-41e6-b0d2-6acfc675711d" (UID: "f7164d67-877d-41e6-b0d2-6acfc675711d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:01.277525 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.277497    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7164d67-877d-41e6-b0d2-6acfc675711d-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:01.277525 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.277521    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:01.277525 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.277530    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrnwb\" (UniqueName: \"kubernetes.io/projected/f7164d67-877d-41e6-b0d2-6acfc675711d-kube-api-access-xrnwb\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:01.277713 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:01.277538    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7164d67-877d-41e6-b0d2-6acfc675711d-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:02.026410 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:02.026332    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"
Sep 17 18:10:02.026410 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:02.026371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9" event={"ID":"f7164d67-877d-41e6-b0d2-6acfc675711d","Type":"ContainerDied","Data":"476ab85308a34198aa7fc7ece68d3d2d469578d7307861847743fc52db4d62b0"}
Sep 17 18:10:02.026913 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:02.026440    2580 scope.go:117] "RemoveContainer" containerID="fa06828f74d2a913ac5c3dbb0382638833485de3e3d4a8f30d604d769a3c46bf"
Sep 17 18:10:02.053843 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:02.053820    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"]
Sep 17 18:10:02.058110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:02.058076    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975zvh9"]
Sep 17 18:10:03.924321 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:03.924290    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:04.024629 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:04.024582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:04.046169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:04.046143    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7164d67-877d-41e6-b0d2-6acfc675711d" path="/var/lib/kubelet/pods/f7164d67-877d-41e6-b0d2-6acfc675711d/volumes"
Sep 17 18:10:06.748936 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.748893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:06.817005 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.816978    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"]
Sep 17 18:10:06.817382 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.817368    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7164d67-877d-41e6-b0d2-6acfc675711d" containerName="main"
Sep 17 18:10:06.817382 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.817383    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7164d67-877d-41e6-b0d2-6acfc675711d" containerName="main"
Sep 17 18:10:06.817470 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.817465    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7164d67-877d-41e6-b0d2-6acfc675711d" containerName="main"
Sep 17 18:10:06.821788 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.821772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.824674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.824629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 18:10:06.824674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.824655    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-vngpg\""
Sep 17 18:10:06.824944 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.824736    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:10:06.824944 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.824873    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:10:06.825102 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.824962    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:10:06.825102 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.825021    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:10:06.837811 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.837791    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"]
Sep 17 18:10:06.840194 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.840172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:06.918909 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.918888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.919034 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.918926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.919034 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.918952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.919034 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.918988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.919034 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.919013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9kf\" (UniqueName: \"kubernetes.io/projected/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kube-api-access-dk9kf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.919294 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.919061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.919294 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.919142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:06.919294 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.919147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:06.919294 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:06.919198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.019775 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.019775 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9kf\" (UniqueName: \"kubernetes.io/projected/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kube-api-access-dk9kf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.019912 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.019912 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.019912 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.019912 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.020152 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.020152 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.019955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.020152 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.020138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.020286 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.020206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.020351 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.020286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.020351 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.020299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.020684 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.020660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.022217 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.022196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.022478 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.022461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.029764 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.029743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9kf\" (UniqueName: \"kubernetes.io/projected/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kube-api-access-dk9kf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.132663 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.132634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:07.149904 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.149867    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:07.420013 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:07.419918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"]
Sep 17 18:10:07.420531 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:10:07.420503    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef647d0_5a41_4bc9_acfb_a2271ab4b6b6.slice/crio-70a7506973ea69d224e95babc66c9564d4762d9ece247a37eafd30325916d910 WatchSource:0}: Error finding container 70a7506973ea69d224e95babc66c9564d4762d9ece247a37eafd30325916d910: Status 404 returned error can't find the container with id 70a7506973ea69d224e95babc66c9564d4762d9ece247a37eafd30325916d910
Sep 17 18:10:08.060999 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:08.060968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" event={"ID":"def647d0-5a41-4bc9-acfb-a2271ab4b6b6","Type":"ContainerStarted","Data":"41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8"}
Sep 17 18:10:08.061466 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:08.061010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" event={"ID":"def647d0-5a41-4bc9-acfb-a2271ab4b6b6","Type":"ContainerStarted","Data":"70a7506973ea69d224e95babc66c9564d4762d9ece247a37eafd30325916d910"}
Sep 17 18:10:08.937366 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:08.937330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:10.075595 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:10.075515    2580 generic.go:356] "Generic (PLEG): container finished" podID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerID="41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8" exitCode=0
Sep 17 18:10:10.075595 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:10.075578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" event={"ID":"def647d0-5a41-4bc9-acfb-a2271ab4b6b6","Type":"ContainerDied","Data":"41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8"}
Sep 17 18:10:11.083978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:11.083943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" event={"ID":"def647d0-5a41-4bc9-acfb-a2271ab4b6b6","Type":"ContainerStarted","Data":"1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53"}
Sep 17 18:10:11.131465 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:11.131412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podStartSLOduration=2.909352022 podStartE2EDuration="5.131396391s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.222044369s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:11.127275704 +0000 UTC m=+1787.466669296" watchObservedRunningTime="2026-09-17 18:10:11.131396391 +0000 UTC m=+1787.470789982"
Sep 17 18:10:11.133155 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:11.133132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:17.133050 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:17.133015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:10:17.134931 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:17.134904    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:10:17.233598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:17.233563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:23.828748 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:23.828638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:23.836909 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:23.836878    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"]
Sep 17 18:10:23.847236 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:10:23.847189    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:10:23.847236 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:23.847236 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:23.847236 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="038ecefbf9e43f54b06e7cd78ba73426d3f1502469b71c5d859a3ed20d14afe1" cmd=["/bin/sleep","15"]
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:23.847255    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" message="" err=<
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:10:23.847271    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" podUID="426d3a06-5eb6-4daf-a65d-effa9f32132e" containerName="main" containerID="cri-o://038ecefbf9e43f54b06e7cd78ba73426d3f1502469b71c5d859a3ed20d14afe1"
Sep 17 18:10:23.847535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:23.847316    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" podUID="426d3a06-5eb6-4daf-a65d-effa9f32132e" containerName="main" containerID="cri-o://038ecefbf9e43f54b06e7cd78ba73426d3f1502469b71c5d859a3ed20d14afe1" gracePeriod=60
Sep 17 18:10:23.894543 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:23.894484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:23.987327 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:23.987289    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" podUID="426d3a06-5eb6-4daf-a65d-effa9f32132e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 17 18:10:24.157864 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.157838    2580 generic.go:356] "Generic (PLEG): container finished" podID="426d3a06-5eb6-4daf-a65d-effa9f32132e" containerID="038ecefbf9e43f54b06e7cd78ba73426d3f1502469b71c5d859a3ed20d14afe1" exitCode=0
Sep 17 18:10:24.158016 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.157921    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" event={"ID":"426d3a06-5eb6-4daf-a65d-effa9f32132e","Type":"ContainerDied","Data":"038ecefbf9e43f54b06e7cd78ba73426d3f1502469b71c5d859a3ed20d14afe1"}
Sep 17 18:10:24.598982 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.598959    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:10:24.773413 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7x9h\" (UniqueName: \"kubernetes.io/projected/426d3a06-5eb6-4daf-a65d-effa9f32132e-kube-api-access-q7x9h\") pod \"426d3a06-5eb6-4daf-a65d-effa9f32132e\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") "
Sep 17 18:10:24.773413 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-tmp-dir\") pod \"426d3a06-5eb6-4daf-a65d-effa9f32132e\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") "
Sep 17 18:10:24.773637 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-home\") pod \"426d3a06-5eb6-4daf-a65d-effa9f32132e\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") "
Sep 17 18:10:24.773637 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-dshm\") pod \"426d3a06-5eb6-4daf-a65d-effa9f32132e\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") "
Sep 17 18:10:24.773637 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426d3a06-5eb6-4daf-a65d-effa9f32132e-tls-certs\") pod \"426d3a06-5eb6-4daf-a65d-effa9f32132e\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") "
Sep 17 18:10:24.773637 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-model-cache\") pod \"426d3a06-5eb6-4daf-a65d-effa9f32132e\" (UID: \"426d3a06-5eb6-4daf-a65d-effa9f32132e\") "
Sep 17 18:10:24.773854 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-home" (OuterVolumeSpecName: "home") pod "426d3a06-5eb6-4daf-a65d-effa9f32132e" (UID: "426d3a06-5eb6-4daf-a65d-effa9f32132e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:24.773854 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "426d3a06-5eb6-4daf-a65d-effa9f32132e" (UID: "426d3a06-5eb6-4daf-a65d-effa9f32132e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:24.773948 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-model-cache" (OuterVolumeSpecName: "model-cache") pod "426d3a06-5eb6-4daf-a65d-effa9f32132e" (UID: "426d3a06-5eb6-4daf-a65d-effa9f32132e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:24.773948 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773873    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:24.773948 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.773894    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:24.775655 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.775628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-dshm" (OuterVolumeSpecName: "dshm") pod "426d3a06-5eb6-4daf-a65d-effa9f32132e" (UID: "426d3a06-5eb6-4daf-a65d-effa9f32132e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:24.775856 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.775834    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426d3a06-5eb6-4daf-a65d-effa9f32132e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "426d3a06-5eb6-4daf-a65d-effa9f32132e" (UID: "426d3a06-5eb6-4daf-a65d-effa9f32132e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:24.775959 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.775943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d3a06-5eb6-4daf-a65d-effa9f32132e-kube-api-access-q7x9h" (OuterVolumeSpecName: "kube-api-access-q7x9h") pod "426d3a06-5eb6-4daf-a65d-effa9f32132e" (UID: "426d3a06-5eb6-4daf-a65d-effa9f32132e"). InnerVolumeSpecName "kube-api-access-q7x9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:24.874803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.874772    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:24.874803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.874794    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426d3a06-5eb6-4daf-a65d-effa9f32132e-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:24.874803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.874806    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426d3a06-5eb6-4daf-a65d-effa9f32132e-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:24.875261 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:24.874814    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7x9h\" (UniqueName: \"kubernetes.io/projected/426d3a06-5eb6-4daf-a65d-effa9f32132e-kube-api-access-q7x9h\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:10:25.165651 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:25.165617    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"
Sep 17 18:10:25.165814 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:25.165644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4" event={"ID":"426d3a06-5eb6-4daf-a65d-effa9f32132e","Type":"ContainerDied","Data":"2e4549bceb5b6b89878752387462fa9980015b8d23de35ff4027240f7fe1bf6e"}
Sep 17 18:10:25.165814 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:25.165708    2580 scope.go:117] "RemoveContainer" containerID="038ecefbf9e43f54b06e7cd78ba73426d3f1502469b71c5d859a3ed20d14afe1"
Sep 17 18:10:25.194628 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:25.194604    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:10:25.198893 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:25.198870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"]
Sep 17 18:10:25.200295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:25.200277    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:10:25.204299 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:25.204275    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l9rv4"]
Sep 17 18:10:26.045886 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:26.045853    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426d3a06-5eb6-4daf-a65d-effa9f32132e" path="/var/lib/kubelet/pods/426d3a06-5eb6-4daf-a65d-effa9f32132e/volumes"
Sep 17 18:10:27.133465 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:27.133427    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:10:30.795699 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:30.795648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:31.869036 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:31.868988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:37.133129 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:37.133068    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:10:37.969107 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:37.969025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:47.134007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:47.133970    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:10:52.569833 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:52.569798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:57.133776 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:10:57.133728    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:11:02.351893 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:02.351855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:02.425673 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:02.425634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:07.134057 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:07.134012    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:11:12.639500 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:12.639448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:12.717892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:12.717829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:17.133808 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:17.133771    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:11:27.133822 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:27.133772    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:11:37.133310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:37.133269    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:11:47.133355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:47.133256    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:11:57.143958 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:57.143922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:11:57.151823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:11:57.151795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:12:01.877795 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:01.877762    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"]
Sep 17 18:12:01.889255 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:01.889225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:01.989399 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:01.989326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:02.707815 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:02.707780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:02.806453 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:02.806423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:09.035985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:09.035949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:14.867600 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:14.867560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:14.929837 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:14.929804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:16.900772 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:16.900704    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" containerID="cri-o://1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53" gracePeriod=45
Sep 17 18:12:17.145151 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.145113    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 18:12:17.648950 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.648924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:12:17.771066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-cabundle-cert\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-home\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771258 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk9kf\" (UniqueName: \"kubernetes.io/projected/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kube-api-access-dk9kf\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771258 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-dshm\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771258 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771148    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tls-certs\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771258 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-model-cache\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771258 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kserve-provision-location\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771258 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tmp-dir\") pod \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\" (UID: \"def647d0-5a41-4bc9-acfb-a2271ab4b6b6\") "
Sep 17 18:12:17.771588 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:17.771588 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771560    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:17.771890 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.771860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-home" (OuterVolumeSpecName: "home") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:17.773457 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.773435    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kube-api-access-dk9kf" (OuterVolumeSpecName: "kube-api-access-dk9kf") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "kube-api-access-dk9kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:17.773547 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.773504    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:17.773547 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.773539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-dshm" (OuterVolumeSpecName: "dshm") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:17.787983 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.787963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:17.791596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.791576    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "def647d0-5a41-4bc9-acfb-a2271ab4b6b6" (UID: "def647d0-5a41-4bc9-acfb-a2271ab4b6b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:17.804699 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.804674    2580 generic.go:356] "Generic (PLEG): container finished" podID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerID="1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53" exitCode=0
Sep 17 18:12:17.804805 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.804718    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"
Sep 17 18:12:17.804805 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.804757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" event={"ID":"def647d0-5a41-4bc9-acfb-a2271ab4b6b6","Type":"ContainerDied","Data":"1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53"}
Sep 17 18:12:17.804892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.804812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs" event={"ID":"def647d0-5a41-4bc9-acfb-a2271ab4b6b6","Type":"ContainerDied","Data":"70a7506973ea69d224e95babc66c9564d4762d9ece247a37eafd30325916d910"}
Sep 17 18:12:17.804892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.804837    2580 scope.go:117] "RemoveContainer" containerID="1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53"
Sep 17 18:12:17.822114 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.822069    2580 scope.go:117] "RemoveContainer" containerID="41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8"
Sep 17 18:12:17.832502 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.832484    2580 scope.go:117] "RemoveContainer" containerID="1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53"
Sep 17 18:12:17.832798 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:12:17.832778    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53\": container with ID starting with 1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53 not found: ID does not exist" containerID="1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53"
Sep 17 18:12:17.832853 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.832805    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53"} err="failed to get container status \"1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53\": rpc error: code = NotFound desc = could not find container \"1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53\": container with ID starting with 1a81c94e1e0d438ffd3ed818d13d3db89498bc8a97fb8661923cc526745eab53 not found: ID does not exist"
Sep 17 18:12:17.832853 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.832821    2580 scope.go:117] "RemoveContainer" containerID="41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8"
Sep 17 18:12:17.833159 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:12:17.833141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8\": container with ID starting with 41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8 not found: ID does not exist" containerID="41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8"
Sep 17 18:12:17.833212 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.833165    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8"} err="failed to get container status \"41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8\": rpc error: code = NotFound desc = could not find container \"41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8\": container with ID starting with 41fa5ec5b37fb3aa5906967dbd79f28d6b5c1c911bc10f73d30e74d259d888e8 not found: ID does not exist"
Sep 17 18:12:17.834591 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.834567    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"]
Sep 17 18:12:17.838059 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.838039    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b9655677flqzqs"]
Sep 17 18:12:17.872210 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872191    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:17.872210 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872209    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:17.872328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872232    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:17.872328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872242    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:17.872328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872250    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:17.872328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872257    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9kf\" (UniqueName: \"kubernetes.io/projected/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-kube-api-access-dk9kf\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:17.872328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872264    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:17.872328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:17.872272    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def647d0-5a41-4bc9-acfb-a2271ab4b6b6-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:12:18.045335 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:18.045286    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" path="/var/lib/kubelet/pods/def647d0-5a41-4bc9-acfb-a2271ab4b6b6/volumes"
Sep 17 18:12:22.892892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.892858    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"]
Sep 17 18:12:22.893310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893220    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426d3a06-5eb6-4daf-a65d-effa9f32132e" containerName="main"
Sep 17 18:12:22.893310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893234    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="426d3a06-5eb6-4daf-a65d-effa9f32132e" containerName="main"
Sep 17 18:12:22.893310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893245    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main"
Sep 17 18:12:22.893310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893252    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main"
Sep 17 18:12:22.893310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893265    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="storage-initializer"
Sep 17 18:12:22.893310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893271    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="storage-initializer"
Sep 17 18:12:22.893535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893319    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="426d3a06-5eb6-4daf-a65d-effa9f32132e" containerName="main"
Sep 17 18:12:22.893535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.893329    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="def647d0-5a41-4bc9-acfb-a2271ab4b6b6" containerName="main"
Sep 17 18:12:22.906329 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.906302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:22.907126 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.907100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"]
Sep 17 18:12:22.910516 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.910492    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-qtjmg\""
Sep 17 18:12:22.910694 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.910508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:12:22.910910 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.910522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:12:22.910910 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.910642    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:12:22.930847 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:22.930795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:23.006720 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.006695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.006823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.006726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fde255-98e5-4410-a214-832dfa36ba87-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.006823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.006750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.006823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.006781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.006823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.006801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djxh\" (UniqueName: \"kubernetes.io/projected/87fde255-98e5-4410-a214-832dfa36ba87-kube-api-access-5djxh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.006977 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.006862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fde255-98e5-4410-a214-832dfa36ba87-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108350 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108533 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djxh\" (UniqueName: \"kubernetes.io/projected/87fde255-98e5-4410-a214-832dfa36ba87-kube-api-access-5djxh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108737 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108862 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.108969 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.108920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.110540 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.110518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.110860 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.110843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fde255-98e5-4410-a214-832dfa36ba87-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.117939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.117919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djxh\" (UniqueName: \"kubernetes.io/projected/87fde255-98e5-4410-a214-832dfa36ba87-kube-api-access-5djxh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.131794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.131766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:23.217333 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.217281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:23.492184 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.492158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"]
Sep 17 18:12:23.493582 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:12:23.493557    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fde255_98e5_4410_a214_832dfa36ba87.slice/crio-161d8b48517b234f667ea592b14ccc87270a296fb54e9175d8f579b42af2dfd8 WatchSource:0}: Error finding container 161d8b48517b234f667ea592b14ccc87270a296fb54e9175d8f579b42af2dfd8: Status 404 returned error can't find the container with id 161d8b48517b234f667ea592b14ccc87270a296fb54e9175d8f579b42af2dfd8
Sep 17 18:12:23.839032 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.838951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" event={"ID":"87fde255-98e5-4410-a214-832dfa36ba87","Type":"ContainerStarted","Data":"2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391"}
Sep 17 18:12:23.839032 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.838998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" event={"ID":"87fde255-98e5-4410-a214-832dfa36ba87","Type":"ContainerStarted","Data":"161d8b48517b234f667ea592b14ccc87270a296fb54e9175d8f579b42af2dfd8"}
Sep 17 18:12:23.863138 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:23.863079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" podStartSLOduration=1.863064185 podStartE2EDuration="1.863064185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:23.862228244 +0000 UTC m=+1920.201621835" watchObservedRunningTime="2026-09-17 18:12:23.863064185 +0000 UTC m=+1920.202457775"
Sep 17 18:12:24.217975 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:24.217944    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:27.658434 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:27.658359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:33.218343 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:33.218306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:33.230622 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:33.230598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:33.903512 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:33.903485    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:12:42.686043 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:42.686000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:51.933674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:51.933640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:52.043884 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:52.043849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:58.337315 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:12:58.337285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:01.688906 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.688870    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"]
Sep 17 18:13:01.692323 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.692303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.695177 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.695154    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 18:13:01.695300 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.695280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 18:13:01.695359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.695313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-86sng\""
Sep 17 18:13:01.695611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.695592    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 18:13:01.704854 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.704833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"]
Sep 17 18:13:01.714468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.714441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:01.781906 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.781885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.782014 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.781914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.782014 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.781941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.782168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.782032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.782168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.782075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.782168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.782134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226xm\" (UniqueName: \"kubernetes.io/projected/45d8f928-0457-4495-8524-790252490994-kube-api-access-226xm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.782301 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.782216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d8f928-0457-4495-8524-790252490994-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.882755 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.882728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d8f928-0457-4495-8524-790252490994-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.882849 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.882779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.882849 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.882798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.882849 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.882820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.882981 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.882860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.882981 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.882886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.882981 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.882912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226xm\" (UniqueName: \"kubernetes.io/projected/45d8f928-0457-4495-8524-790252490994-kube-api-access-226xm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.883195 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.883170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.883333 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.883311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.883426 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.883332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.883426 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.883363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.885342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.885320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.885430 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.885416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d8f928-0457-4495-8524-790252490994-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.890481 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.890458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-226xm\" (UniqueName: \"kubernetes.io/projected/45d8f928-0457-4495-8524-790252490994-kube-api-access-226xm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq\" (UID: \"45d8f928-0457-4495-8524-790252490994\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:01.973030 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:01.972957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:02.003215 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:02.003189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:02.284240 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:02.284212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"]
Sep 17 18:13:02.327907 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:02.327885    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:13:03.064896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:03.064860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerStarted","Data":"03355674d13c73d0a32ca1dc3ed87efd1e17d212f7c7dedc83bbb2f6929e4b6f"}
Sep 17 18:13:04.477703 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:04.477668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:04.578931 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:04.578862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:08.852710 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:08.852669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:13.121432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:13.121356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerStarted","Data":"2776e0a25578a842a7ac47e6bccec1ff5b1dcd92f7783bda7a4c71bd90d7af08"}
Sep 17 18:13:14.127963 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:14.127930    2580 generic.go:356] "Generic (PLEG): container finished" podID="45d8f928-0457-4495-8524-790252490994" containerID="2776e0a25578a842a7ac47e6bccec1ff5b1dcd92f7783bda7a4c71bd90d7af08" exitCode=0
Sep 17 18:13:14.128335 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:14.128013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerDied","Data":"2776e0a25578a842a7ac47e6bccec1ff5b1dcd92f7783bda7a4c71bd90d7af08"}
Sep 17 18:13:15.135210 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:15.135120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerStarted","Data":"7ed728ab113897041edf3041acf15fbef638baf932cc9fd6409310b6d2296deb"}
Sep 17 18:13:15.135210 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:15.135182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerStarted","Data":"247a670fa526a4d7bbafabbb93d057dfe846c9b3bc12512380e78ce3c84e99d2"}
Sep 17 18:13:15.158269 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:15.158225    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podStartSLOduration=2.626760501 podStartE2EDuration="14.158208723s" totalImagesPullingTime="10.344782935s" totalInitContainerRuntime="1.186665287s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:15.155891642 +0000 UTC m=+1971.495285231" watchObservedRunningTime="2026-09-17 18:13:15.158208723 +0000 UTC m=+1971.497602313"
Sep 17 18:13:16.003942 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:16.003909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:22.003608 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:22.003573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:13:22.005348 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:22.005319    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:13:32.003735 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:32.003687    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:13:42.003691 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:42.003650    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:13:52.003844 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:13:52.003803    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:14:02.003743 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:02.003660    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:14:11.230318 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:11.230281    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:11.428978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:11.428942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:12.004195 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:12.004153    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:14:18.151201 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:18.151156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:22.003884 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:22.003840    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:14:26.775099 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:26.775044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:32.004441 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:32.004402    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:14:42.003875 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:42.003831    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:14:45.368728 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:45.368614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:45.468528 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:45.468488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:52.004432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:52.004388    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:14:54.060573 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:14:54.060536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:02.013960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:02.013885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:15:02.021610 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:02.021584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:15:06.223835 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:06.223783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:06.226200 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:06.226167    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"]
Sep 17 18:15:06.226625 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:06.226573    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="modelcar" containerID="cri-o://7ed728ab113897041edf3041acf15fbef638baf932cc9fd6409310b6d2296deb" gracePeriod=60
Sep 17 18:15:06.301733 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:06.301695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:06.772344 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:06.772319    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq_45d8f928-0457-4495-8524-790252490994/modelcar/0.log"
Sep 17 18:15:06.772960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:06.772938    2580 generic.go:356] "Generic (PLEG): container finished" podID="45d8f928-0457-4495-8524-790252490994" containerID="7ed728ab113897041edf3041acf15fbef638baf932cc9fd6409310b6d2296deb" exitCode=143
Sep 17 18:15:06.773068 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:06.772998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerDied","Data":"7ed728ab113897041edf3041acf15fbef638baf932cc9fd6409310b6d2296deb"}
Sep 17 18:15:21.250366 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:21.250304    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" containerID="cri-o://247a670fa526a4d7bbafabbb93d057dfe846c9b3bc12512380e78ce3c84e99d2" gracePeriod=45
Sep 17 18:15:21.864989 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:21.864966    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq_45d8f928-0457-4495-8524-790252490994/modelcar/0.log"
Sep 17 18:15:21.865665 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:21.865644    2580 generic.go:356] "Generic (PLEG): container finished" podID="45d8f928-0457-4495-8524-790252490994" containerID="247a670fa526a4d7bbafabbb93d057dfe846c9b3bc12512380e78ce3c84e99d2" exitCode=0
Sep 17 18:15:21.865741 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:21.865720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerDied","Data":"247a670fa526a4d7bbafabbb93d057dfe846c9b3bc12512380e78ce3c84e99d2"}
Sep 17 18:15:21.899482 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:21.899457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:21.958240 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:21.958191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:22.015018 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.014944    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 18:15:22.498022 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.498001    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq_45d8f928-0457-4495-8524-790252490994/modelcar/0.log"
Sep 17 18:15:22.500762 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.500745    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:15:22.516642 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516622    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-model-cache\") pod \"45d8f928-0457-4495-8524-790252490994\" (UID: \"45d8f928-0457-4495-8524-790252490994\") "
Sep 17 18:15:22.516705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-home\") pod \"45d8f928-0457-4495-8524-790252490994\" (UID: \"45d8f928-0457-4495-8524-790252490994\") "
Sep 17 18:15:22.516705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-dshm\") pod \"45d8f928-0457-4495-8524-790252490994\" (UID: \"45d8f928-0457-4495-8524-790252490994\") "
Sep 17 18:15:22.516779 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-kserve-provision-location\") pod \"45d8f928-0457-4495-8524-790252490994\" (UID: \"45d8f928-0457-4495-8524-790252490994\") "
Sep 17 18:15:22.516779 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-tmp-dir\") pod \"45d8f928-0457-4495-8524-790252490994\" (UID: \"45d8f928-0457-4495-8524-790252490994\") "
Sep 17 18:15:22.516779 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d8f928-0457-4495-8524-790252490994-tls-certs\") pod \"45d8f928-0457-4495-8524-790252490994\" (UID: \"45d8f928-0457-4495-8524-790252490994\") "
Sep 17 18:15:22.516933 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-model-cache" (OuterVolumeSpecName: "model-cache") pod "45d8f928-0457-4495-8524-790252490994" (UID: "45d8f928-0457-4495-8524-790252490994"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:22.516933 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.516809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-226xm\" (UniqueName: \"kubernetes.io/projected/45d8f928-0457-4495-8524-790252490994-kube-api-access-226xm\") pod \"45d8f928-0457-4495-8524-790252490994\" (UID: \"45d8f928-0457-4495-8524-790252490994\") "
Sep 17 18:15:22.517066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.517041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45d8f928-0457-4495-8524-790252490994" (UID: "45d8f928-0457-4495-8524-790252490994"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:22.517166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.517147    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.517221 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.517171    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.517369 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.517342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-home" (OuterVolumeSpecName: "home") pod "45d8f928-0457-4495-8524-790252490994" (UID: "45d8f928-0457-4495-8524-790252490994"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:22.518941 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.518914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-dshm" (OuterVolumeSpecName: "dshm") pod "45d8f928-0457-4495-8524-790252490994" (UID: "45d8f928-0457-4495-8524-790252490994"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:22.519029 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.518949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d8f928-0457-4495-8524-790252490994-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45d8f928-0457-4495-8524-790252490994" (UID: "45d8f928-0457-4495-8524-790252490994"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:22.519029 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.518995    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d8f928-0457-4495-8524-790252490994-kube-api-access-226xm" (OuterVolumeSpecName: "kube-api-access-226xm") pod "45d8f928-0457-4495-8524-790252490994" (UID: "45d8f928-0457-4495-8524-790252490994"). InnerVolumeSpecName "kube-api-access-226xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:22.531208 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.531175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45d8f928-0457-4495-8524-790252490994" (UID: "45d8f928-0457-4495-8524-790252490994"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:22.618049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.617993    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.618049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.618016    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.618049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.618024    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d8f928-0457-4495-8524-790252490994-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.618049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.618031    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d8f928-0457-4495-8524-790252490994-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.618049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.618041    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226xm\" (UniqueName: \"kubernetes.io/projected/45d8f928-0457-4495-8524-790252490994-kube-api-access-226xm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.873254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.873192    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq_45d8f928-0457-4495-8524-790252490994/modelcar/0.log"
Sep 17 18:15:22.873967 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.873945    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"
Sep 17 18:15:22.874072 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.873962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq" event={"ID":"45d8f928-0457-4495-8524-790252490994","Type":"ContainerDied","Data":"03355674d13c73d0a32ca1dc3ed87efd1e17d212f7c7dedc83bbb2f6929e4b6f"}
Sep 17 18:15:22.874072 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.874048    2580 scope.go:117] "RemoveContainer" containerID="7ed728ab113897041edf3041acf15fbef638baf932cc9fd6409310b6d2296deb"
Sep 17 18:15:22.883888 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.883868    2580 scope.go:117] "RemoveContainer" containerID="247a670fa526a4d7bbafabbb93d057dfe846c9b3bc12512380e78ce3c84e99d2"
Sep 17 18:15:22.891716 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.891699    2580 scope.go:117] "RemoveContainer" containerID="2776e0a25578a842a7ac47e6bccec1ff5b1dcd92f7783bda7a4c71bd90d7af08"
Sep 17 18:15:22.895466 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.895444    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"]
Sep 17 18:15:22.900045 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:22.900020    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dqwvcq"]
Sep 17 18:15:24.046170 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:24.046135    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d8f928-0457-4495-8524-790252490994" path="/var/lib/kubelet/pods/45d8f928-0457-4495-8524-790252490994/volumes"
Sep 17 18:15:25.232039 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:25.232015    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:15:25.242839 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:25.242815    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:15:35.197588 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.197551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:35.206934 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.206905    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"]
Sep 17 18:15:35.216134 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:15:35.216100    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:15:35.216134 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:35.216134 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:35.216134 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391" cmd=["/bin/sleep","15"]
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.216152    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" message="" err=<
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:15:35.216164    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" podUID="87fde255-98e5-4410-a214-832dfa36ba87" containerName="main" containerID="cri-o://2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391"
Sep 17 18:15:35.216409 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.216197    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" podUID="87fde255-98e5-4410-a214-832dfa36ba87" containerName="main" containerID="cri-o://2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391" gracePeriod=60
Sep 17 18:15:35.294763 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.294720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:35.564181 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.564159    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:15:35.611462 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611436    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-model-cache\") pod \"87fde255-98e5-4410-a214-832dfa36ba87\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") "
Sep 17 18:15:35.611626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5djxh\" (UniqueName: \"kubernetes.io/projected/87fde255-98e5-4410-a214-832dfa36ba87-kube-api-access-5djxh\") pod \"87fde255-98e5-4410-a214-832dfa36ba87\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") "
Sep 17 18:15:35.611626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-dshm\") pod \"87fde255-98e5-4410-a214-832dfa36ba87\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") "
Sep 17 18:15:35.611626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-home\") pod \"87fde255-98e5-4410-a214-832dfa36ba87\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") "
Sep 17 18:15:35.611626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611575    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fde255-98e5-4410-a214-832dfa36ba87-tls-certs\") pod \"87fde255-98e5-4410-a214-832dfa36ba87\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") "
Sep 17 18:15:35.611626 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-tmp-dir\") pod \"87fde255-98e5-4410-a214-832dfa36ba87\" (UID: \"87fde255-98e5-4410-a214-832dfa36ba87\") "
Sep 17 18:15:35.611922 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-model-cache" (OuterVolumeSpecName: "model-cache") pod "87fde255-98e5-4410-a214-832dfa36ba87" (UID: "87fde255-98e5-4410-a214-832dfa36ba87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:35.611983 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87fde255-98e5-4410-a214-832dfa36ba87" (UID: "87fde255-98e5-4410-a214-832dfa36ba87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:35.611983 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.611934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-home" (OuterVolumeSpecName: "home") pod "87fde255-98e5-4410-a214-832dfa36ba87" (UID: "87fde255-98e5-4410-a214-832dfa36ba87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:35.613699 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.613671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-dshm" (OuterVolumeSpecName: "dshm") pod "87fde255-98e5-4410-a214-832dfa36ba87" (UID: "87fde255-98e5-4410-a214-832dfa36ba87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:35.613799 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.613720    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fde255-98e5-4410-a214-832dfa36ba87-kube-api-access-5djxh" (OuterVolumeSpecName: "kube-api-access-5djxh") pod "87fde255-98e5-4410-a214-832dfa36ba87" (UID: "87fde255-98e5-4410-a214-832dfa36ba87"). InnerVolumeSpecName "kube-api-access-5djxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:35.613799 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.613725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fde255-98e5-4410-a214-832dfa36ba87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87fde255-98e5-4410-a214-832dfa36ba87" (UID: "87fde255-98e5-4410-a214-832dfa36ba87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:35.712886 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.712852    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fde255-98e5-4410-a214-832dfa36ba87-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.712886 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.712883    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.713078 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.712894    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.713078 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.712907    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djxh\" (UniqueName: \"kubernetes.io/projected/87fde255-98e5-4410-a214-832dfa36ba87-kube-api-access-5djxh\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.713078 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.712919    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.713078 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.712929    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fde255-98e5-4410-a214-832dfa36ba87-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.940776 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.940739    2580 generic.go:356] "Generic (PLEG): container finished" podID="87fde255-98e5-4410-a214-832dfa36ba87" containerID="2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391" exitCode=0
Sep 17 18:15:35.940919 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.940788    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"
Sep 17 18:15:35.940919 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.940822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" event={"ID":"87fde255-98e5-4410-a214-832dfa36ba87","Type":"ContainerDied","Data":"2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391"}
Sep 17 18:15:35.940919 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.940879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg" event={"ID":"87fde255-98e5-4410-a214-832dfa36ba87","Type":"ContainerDied","Data":"161d8b48517b234f667ea592b14ccc87270a296fb54e9175d8f579b42af2dfd8"}
Sep 17 18:15:35.940919 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.940903    2580 scope.go:117] "RemoveContainer" containerID="2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391"
Sep 17 18:15:35.950498 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.950480    2580 scope.go:117] "RemoveContainer" containerID="2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391"
Sep 17 18:15:35.950757 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:15:35.950737    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391\": container with ID starting with 2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391 not found: ID does not exist" containerID="2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391"
Sep 17 18:15:35.950826 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.950763    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391"} err="failed to get container status \"2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391\": rpc error: code = NotFound desc = could not find container \"2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391\": container with ID starting with 2e1ca8cfd1a8747618f0ba1ecffb7e519e2e013de37b53403ae4ae1a1e96d391 not found: ID does not exist"
Sep 17 18:15:35.963198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.963171    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"]
Sep 17 18:15:35.966588 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:35.966570    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969dnqmg"]
Sep 17 18:15:36.045442 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:36.045420    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fde255-98e5-4410-a214-832dfa36ba87" path="/var/lib/kubelet/pods/87fde255-98e5-4410-a214-832dfa36ba87/volumes"
Sep 17 18:15:56.350098 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:56.350047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:56.447799 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:15:56.447766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:02.220032 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:02.220002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:07.974416 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:07.974384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:08.070052 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:08.070026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:11.506176 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506140    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"]
Sep 17 18:16:11.506517 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506500    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main"
Sep 17 18:16:11.506517 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506512    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main"
Sep 17 18:16:11.506593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506525    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d8f928-0457-4495-8524-790252490994" containerName="modelcar-init"
Sep 17 18:16:11.506593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506530    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d8f928-0457-4495-8524-790252490994" containerName="modelcar-init"
Sep 17 18:16:11.506593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506544    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fde255-98e5-4410-a214-832dfa36ba87" containerName="main"
Sep 17 18:16:11.506593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506550    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fde255-98e5-4410-a214-832dfa36ba87" containerName="main"
Sep 17 18:16:11.506593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506556    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d8f928-0457-4495-8524-790252490994" containerName="modelcar"
Sep 17 18:16:11.506593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506561    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d8f928-0457-4495-8524-790252490994" containerName="modelcar"
Sep 17 18:16:11.506787 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506617    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fde255-98e5-4410-a214-832dfa36ba87" containerName="main"
Sep 17 18:16:11.506787 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506625    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d8f928-0457-4495-8524-790252490994" containerName="main"
Sep 17 18:16:11.506787 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.506634    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d8f928-0457-4495-8524-790252490994" containerName="modelcar"
Sep 17 18:16:11.509350 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.509332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.512218 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.512197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 18:16:11.512369 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.512231    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 18:16:11.512369 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.512297    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-2mv88\""
Sep 17 18:16:11.512556 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.512539    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 18:16:11.512616 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.512541    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 18:16:11.512659 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.512546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:16:11.519368 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.519348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"]
Sep 17 18:16:11.539495 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.539461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:11.573242 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-home\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.573358 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-model-cache\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.573358 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4eb39a-3433-4540-b559-90b4847568fe-tls-certs\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.573358 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl54h\" (UniqueName: \"kubernetes.io/projected/4a4eb39a-3433-4540-b559-90b4847568fe-kube-api-access-pl54h\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.573358 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.573538 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4eb39a-3433-4540-b559-90b4847568fe-cabundle-cert\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.573538 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-tmp-dir\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.573538 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.573415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-dshm\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673701 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-tmp-dir\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-dshm\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-home\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-model-cache\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4eb39a-3433-4540-b559-90b4847568fe-tls-certs\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl54h\" (UniqueName: \"kubernetes.io/projected/4a4eb39a-3433-4540-b559-90b4847568fe-kube-api-access-pl54h\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.673974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.673864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4eb39a-3433-4540-b559-90b4847568fe-cabundle-cert\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.674177 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.674157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-tmp-dir\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.674177 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.674172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-home\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.674299 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.674256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-model-cache\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.674516 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.674496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.674594 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.674538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4eb39a-3433-4540-b559-90b4847568fe-cabundle-cert\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.676101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.676072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-dshm\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.676388 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.676371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4eb39a-3433-4540-b559-90b4847568fe-tls-certs\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.691776 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.691753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl54h\" (UniqueName: \"kubernetes.io/projected/4a4eb39a-3433-4540-b559-90b4847568fe-kube-api-access-pl54h\") pod \"custom-route-timeout-test-kserve-ff84bdb94-c5vqd\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:11.789839 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.789754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:11.820303 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:11.820277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:12.103257 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:12.103228    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"]
Sep 17 18:16:12.104373 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:16:12.104350    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4eb39a_3433_4540_b559_90b4847568fe.slice/crio-ebea8bb65cdb80cb87c1ae98d5de0e38fcfdb250df4a3ebb029878e822dfbcb7 WatchSource:0}: Error finding container ebea8bb65cdb80cb87c1ae98d5de0e38fcfdb250df4a3ebb029878e822dfbcb7: Status 404 returned error can't find the container with id ebea8bb65cdb80cb87c1ae98d5de0e38fcfdb250df4a3ebb029878e822dfbcb7
Sep 17 18:16:12.128256 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:12.128232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" event={"ID":"4a4eb39a-3433-4540-b559-90b4847568fe","Type":"ContainerStarted","Data":"ebea8bb65cdb80cb87c1ae98d5de0e38fcfdb250df4a3ebb029878e822dfbcb7"}
Sep 17 18:16:13.146022 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:13.145983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" event={"ID":"4a4eb39a-3433-4540-b559-90b4847568fe","Type":"ContainerStarted","Data":"349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb"}
Sep 17 18:16:15.165005 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:15.164977    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a4eb39a-3433-4540-b559-90b4847568fe" containerID="349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb" exitCode=0
Sep 17 18:16:15.165356 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:15.165048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" event={"ID":"4a4eb39a-3433-4540-b559-90b4847568fe","Type":"ContainerDied","Data":"349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb"}
Sep 17 18:16:16.172671 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:16.172633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" event={"ID":"4a4eb39a-3433-4540-b559-90b4847568fe","Type":"ContainerStarted","Data":"9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e"}
Sep 17 18:16:16.194654 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:16.194595    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podStartSLOduration=3.103563738 podStartE2EDuration="5.194574387s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.091010649s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:16.193276068 +0000 UTC m=+2152.532669680" watchObservedRunningTime="2026-09-17 18:16:16.194574387 +0000 UTC m=+2152.533967963"
Sep 17 18:16:16.821215 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:16.821187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:19.783836 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:19.783797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:21.821236 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:21.821203    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:16:21.822836 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:21.822809    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:16:25.286960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:25.286919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:25.384745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:25.384661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:27.876916 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.876878    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"]
Sep 17 18:16:27.880394 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.880375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:27.883491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.883469    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 18:16:27.883491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.883478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:16:27.883687 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.883673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 18:16:27.883752 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.883722    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 18:16:27.883907 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.883886    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-fn5bf\""
Sep 17 18:16:27.884001 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.883923    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 18:16:27.890635 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.890613    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"]
Sep 17 18:16:27.905902 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:27.905875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:28.014737 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.014699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.014737 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.014737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.014974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.014772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.014974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.014862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.014974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.014902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.015168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.015005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrld\" (UniqueName: \"kubernetes.io/projected/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kube-api-access-hgrld\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.015168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.015042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.015168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.015067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.095716 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.095674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:28.115782 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrld\" (UniqueName: \"kubernetes.io/projected/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kube-api-access-hgrld\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.115928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.115928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.115928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.115928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.115928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.116217 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.116217 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.115960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.116326 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.116225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.116379 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.116334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.116432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.116382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.116491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.116429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.116830 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.116809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.118172 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.118153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.118502 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.118485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.125634 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.125593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrld\" (UniqueName: \"kubernetes.io/projected/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kube-api-access-hgrld\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.191744 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.191717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:28.476977 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:28.476949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"]
Sep 17 18:16:28.478575 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:16:28.478546    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7f9f35_47af_4368_81ce_ff9d9ed3c536.slice/crio-824c9cdebffb267d517f2f8b1dfc69b9fe2ba84e0119d4735c2870466c743c56 WatchSource:0}: Error finding container 824c9cdebffb267d517f2f8b1dfc69b9fe2ba84e0119d4735c2870466c743c56: Status 404 returned error can't find the container with id 824c9cdebffb267d517f2f8b1dfc69b9fe2ba84e0119d4735c2870466c743c56
Sep 17 18:16:29.241506 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:29.241467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" event={"ID":"8f7f9f35-47af-4368-81ce-ff9d9ed3c536","Type":"ContainerStarted","Data":"24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15"}
Sep 17 18:16:29.241992 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:29.241520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" event={"ID":"8f7f9f35-47af-4368-81ce-ff9d9ed3c536","Type":"ContainerStarted","Data":"824c9cdebffb267d517f2f8b1dfc69b9fe2ba84e0119d4735c2870466c743c56"}
Sep 17 18:16:31.785375 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:31.785342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:31.821455 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:31.821413    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:16:32.260948 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:32.260914    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerID="24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15" exitCode=0
Sep 17 18:16:32.261168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:32.260985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" event={"ID":"8f7f9f35-47af-4368-81ce-ff9d9ed3c536","Type":"ContainerDied","Data":"24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15"}
Sep 17 18:16:33.268807 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:33.268757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" event={"ID":"8f7f9f35-47af-4368-81ce-ff9d9ed3c536","Type":"ContainerStarted","Data":"03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143"}
Sep 17 18:16:33.293325 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:33.293252    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podStartSLOduration=3.528633673 podStartE2EDuration="6.29323207s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.764598397s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:33.290550456 +0000 UTC m=+2169.629944046" watchObservedRunningTime="2026-09-17 18:16:33.29323207 +0000 UTC m=+2169.632625663"
Sep 17 18:16:34.191838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:34.191804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:37.516309 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:37.516273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:37.629313 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:37.629276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:38.192637 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:38.192603    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:16:38.194426 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:38.194392    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:16:41.820895 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:41.820846    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:16:45.200339 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:45.200298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:48.192331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:48.192284    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:16:51.820706 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:51.820660    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:16:58.192207 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:16:58.192160    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:17:01.821168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:01.821120    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:17:08.192824 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:08.192779    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:17:11.821412 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:11.821365    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:17:18.192896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:18.192857    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:17:21.821143 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:21.821104    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:17:28.192100 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:28.192025    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:17:31.821488 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:31.821446    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:17:38.192537 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:38.192496    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:17:41.820717 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:41.820673    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:17:48.203237 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:48.203161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:17:48.211169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:48.211150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:17:51.820872 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:17:51.820828    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:18:01.820759 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:01.820714    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:18:05.028012 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:05.027975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:11.821198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:11.821148    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:18:21.835483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:21.835446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:18:21.846238 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:21.846206    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:18:23.712674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:23.712639    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"]
Sep 17 18:18:23.713146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:23.712927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:23.811802 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:23.811766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:26.513848 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:26.513815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:26.521550 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:26.521522    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"]
Sep 17 18:18:26.609714 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:26.609669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:38.738773 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:38.738729    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" containerID="cri-o://03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143" gracePeriod=45
Sep 17 18:18:39.204303 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.204266    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 18:18:39.385172 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.385146    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:18:39.457910 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.457831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgrld\" (UniqueName: \"kubernetes.io/projected/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kube-api-access-hgrld\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.457910 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.457884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-dshm\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.458146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.457996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tls-certs\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.458146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tmp-dir\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.458146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-cabundle-cert\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.458146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-model-cache\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.458146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kserve-provision-location\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.458419 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-home\") pod \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\" (UID: \"8f7f9f35-47af-4368-81ce-ff9d9ed3c536\") "
Sep 17 18:18:39.458419 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458376    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:39.458419 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:39.458741 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.458711    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-home" (OuterVolumeSpecName: "home") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:39.460237 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.460212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:39.460336 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.460218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-dshm" (OuterVolumeSpecName: "dshm") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:39.460380 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.460340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kube-api-access-hgrld" (OuterVolumeSpecName: "kube-api-access-hgrld") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "kube-api-access-hgrld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:39.470178 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.470075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:39.485747 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.485709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f7f9f35-47af-4368-81ce-ff9d9ed3c536" (UID: "8f7f9f35-47af-4368-81ce-ff9d9ed3c536"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:39.559421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559391    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:39.559421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559415    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:39.559565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559427    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgrld\" (UniqueName: \"kubernetes.io/projected/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-kube-api-access-hgrld\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:39.559565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559436    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:39.559565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559444    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:39.559565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559451    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:39.559565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559458    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:39.559565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:39.559465    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f7f9f35-47af-4368-81ce-ff9d9ed3c536-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:40.058355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.058320    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerID="03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143" exitCode=0
Sep 17 18:18:40.058656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.058370    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"
Sep 17 18:18:40.058656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.058394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" event={"ID":"8f7f9f35-47af-4368-81ce-ff9d9ed3c536","Type":"ContainerDied","Data":"03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143"}
Sep 17 18:18:40.058656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.058447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm" event={"ID":"8f7f9f35-47af-4368-81ce-ff9d9ed3c536","Type":"ContainerDied","Data":"824c9cdebffb267d517f2f8b1dfc69b9fe2ba84e0119d4735c2870466c743c56"}
Sep 17 18:18:40.058656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.058472    2580 scope.go:117] "RemoveContainer" containerID="03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143"
Sep 17 18:18:40.067262 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.067246    2580 scope.go:117] "RemoveContainer" containerID="24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15"
Sep 17 18:18:40.075883 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.075864    2580 scope.go:117] "RemoveContainer" containerID="03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143"
Sep 17 18:18:40.076458 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:18:40.076401    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143\": container with ID starting with 03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143 not found: ID does not exist" containerID="03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143"
Sep 17 18:18:40.076458 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.076436    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143"} err="failed to get container status \"03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143\": rpc error: code = NotFound desc = could not find container \"03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143\": container with ID starting with 03f96550cf5220b80f3a0fe51b79437e868c0789cae75536d0c71f5ad105e143 not found: ID does not exist"
Sep 17 18:18:40.076458 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.076458    2580 scope.go:117] "RemoveContainer" containerID="24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15"
Sep 17 18:18:40.076781 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:18:40.076758    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15\": container with ID starting with 24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15 not found: ID does not exist" containerID="24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15"
Sep 17 18:18:40.076825 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.076789    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15"} err="failed to get container status \"24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15\": rpc error: code = NotFound desc = could not find container \"24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15\": container with ID starting with 24fdd1b114d2097c60d2005aed62c951d8ee1b9287f0cbd39c79fe39415f5a15 not found: ID does not exist"
Sep 17 18:18:40.077484 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.077467    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"]
Sep 17 18:18:40.081294 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:40.081274    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b9f6664bf-v6kvm"]
Sep 17 18:18:41.546740 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:41.546702    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" containerID="cri-o://9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e" gracePeriod=45
Sep 17 18:18:41.836918 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:41.836838    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 17 18:18:42.052790 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.052755    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" path="/var/lib/kubelet/pods/8f7f9f35-47af-4368-81ce-ff9d9ed3c536/volumes"
Sep 17 18:18:42.297162 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.297141    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:18:42.380619 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-home\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.380619 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380597    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-dshm\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.380805 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380634    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4eb39a-3433-4540-b559-90b4847568fe-cabundle-cert\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.380805 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-model-cache\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.380805 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380682    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl54h\" (UniqueName: \"kubernetes.io/projected/4a4eb39a-3433-4540-b559-90b4847568fe-kube-api-access-pl54h\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.380960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-kserve-provision-location\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.380960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-tmp-dir\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.380960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4eb39a-3433-4540-b559-90b4847568fe-tls-certs\") pod \"4a4eb39a-3433-4540-b559-90b4847568fe\" (UID: \"4a4eb39a-3433-4540-b559-90b4847568fe\") "
Sep 17 18:18:42.381133 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380967    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:42.381411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.380974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4eb39a-3433-4540-b559-90b4847568fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:42.381411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.381235    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.381411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.381370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-home" (OuterVolumeSpecName: "home") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:42.382820 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.382797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-dshm" (OuterVolumeSpecName: "dshm") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:42.383284 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.383262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4eb39a-3433-4540-b559-90b4847568fe-kube-api-access-pl54h" (OuterVolumeSpecName: "kube-api-access-pl54h") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "kube-api-access-pl54h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:42.383383 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.383304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4eb39a-3433-4540-b559-90b4847568fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:42.387935 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.387912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:42.400728 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.400705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a4eb39a-3433-4540-b559-90b4847568fe" (UID: "4a4eb39a-3433-4540-b559-90b4847568fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:42.482159 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.482140    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.482159 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.482158    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.482266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.482167    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4eb39a-3433-4540-b559-90b4847568fe-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.482266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.482176    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl54h\" (UniqueName: \"kubernetes.io/projected/4a4eb39a-3433-4540-b559-90b4847568fe-kube-api-access-pl54h\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.482266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.482184    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.482266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.482191    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4eb39a-3433-4540-b559-90b4847568fe-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.482266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:42.482199    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4eb39a-3433-4540-b559-90b4847568fe-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:18:43.076098 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.076049    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a4eb39a-3433-4540-b559-90b4847568fe" containerID="9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e" exitCode=0
Sep 17 18:18:43.076580 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.076127    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"
Sep 17 18:18:43.076580 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.076139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" event={"ID":"4a4eb39a-3433-4540-b559-90b4847568fe","Type":"ContainerDied","Data":"9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e"}
Sep 17 18:18:43.076580 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.076195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd" event={"ID":"4a4eb39a-3433-4540-b559-90b4847568fe","Type":"ContainerDied","Data":"ebea8bb65cdb80cb87c1ae98d5de0e38fcfdb250df4a3ebb029878e822dfbcb7"}
Sep 17 18:18:43.076580 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.076218    2580 scope.go:117] "RemoveContainer" containerID="9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e"
Sep 17 18:18:43.087118 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.086995    2580 scope.go:117] "RemoveContainer" containerID="349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb"
Sep 17 18:18:43.095441 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.095423    2580 scope.go:117] "RemoveContainer" containerID="9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e"
Sep 17 18:18:43.095712 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:18:43.095691    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e\": container with ID starting with 9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e not found: ID does not exist" containerID="9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e"
Sep 17 18:18:43.095754 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.095721    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e"} err="failed to get container status \"9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e\": rpc error: code = NotFound desc = could not find container \"9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e\": container with ID starting with 9fc4f630147ad57d92bd7038a50b8039b4f1e7d3773b51582fdf94e41995499e not found: ID does not exist"
Sep 17 18:18:43.095754 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.095737    2580 scope.go:117] "RemoveContainer" containerID="349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb"
Sep 17 18:18:43.095981 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:18:43.095957    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb\": container with ID starting with 349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb not found: ID does not exist" containerID="349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb"
Sep 17 18:18:43.096074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.095983    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb"} err="failed to get container status \"349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb\": rpc error: code = NotFound desc = could not find container \"349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb\": container with ID starting with 349768fb23ef419170aca65c22b5ffa41ec13a9f4e753eb7628f897205164deb not found: ID does not exist"
Sep 17 18:18:43.099398 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.099377    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"]
Sep 17 18:18:43.104362 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:43.104339    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-ff84bdb94-c5vqd"]
Sep 17 18:18:44.046211 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:18:44.046180    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" path="/var/lib/kubelet/pods/4a4eb39a-3433-4540-b559-90b4847568fe/volumes"
Sep 17 18:19:13.816406 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:13.816376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:13.909738 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:13.909709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:16.634625 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:16.634545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:16.737551 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:16.737517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:16.857536 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:16.857508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:16.956611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:16.956578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:20.650619 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:20.650584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:20.751350 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:20.751305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:23.170241 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:23.170212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:26.552499 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:26.552467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:26.652689 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:26.652656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:28.767217 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:28.767173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:28.829647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:28.829614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:40.498991 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.498955    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"]
Sep 17 18:19:40.499435 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499415    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="storage-initializer"
Sep 17 18:19:40.499503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499439    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="storage-initializer"
Sep 17 18:19:40.499503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499462    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main"
Sep 17 18:19:40.499503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499470    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main"
Sep 17 18:19:40.499503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499484    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="storage-initializer"
Sep 17 18:19:40.499503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499493    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="storage-initializer"
Sep 17 18:19:40.499734 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499515    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main"
Sep 17 18:19:40.499734 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499522    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main"
Sep 17 18:19:40.499734 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499605    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7f9f35-47af-4368-81ce-ff9d9ed3c536" containerName="main"
Sep 17 18:19:40.499734 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.499620    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4eb39a-3433-4540-b559-90b4847568fe" containerName="main"
Sep 17 18:19:40.502861 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.502841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.505438 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.505419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 18:19:40.505521 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.505421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-rk9dn\""
Sep 17 18:19:40.505521 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.505420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:19:40.505613 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.505428    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:19:40.506018 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.505994    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:19:40.506141 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.506108    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:19:40.516886 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.516863    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"]
Sep 17 18:19:40.595405 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.595374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:40.688846 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.688821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.689004 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.688861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.689004 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.688888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.689004 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.688910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rt5\" (UniqueName: \"kubernetes.io/projected/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kube-api-access-74rt5\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.689004 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.688973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.689224 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.689003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d0dca22-7f6a-43bd-aa4f-600ad786b445-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.689224 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.689040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.689224 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.689065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.778424 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.778344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:40.789690 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.789669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.789811 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.789702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d0dca22-7f6a-43bd-aa4f-600ad786b445-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.789811 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.789725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.789975 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.789955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790039 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790122 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790122 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790122 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790290 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790290 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rt5\" (UniqueName: \"kubernetes.io/projected/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kube-api-access-74rt5\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790290 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790460 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.790527 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.790507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d0dca22-7f6a-43bd-aa4f-600ad786b445-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.792279 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.792246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.792483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.792463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.798749 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.798727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rt5\" (UniqueName: \"kubernetes.io/projected/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kube-api-access-74rt5\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:40.813495 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:40.813475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:41.089450 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:41.089418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"]
Sep 17 18:19:41.090617 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:19:41.090584    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0dca22_7f6a_43bd_aa4f_600ad786b445.slice/crio-b623195e2605f7c1401a25807c925beedc104268409b494aae4efececd755e8c WatchSource:0}: Error finding container b623195e2605f7c1401a25807c925beedc104268409b494aae4efececd755e8c: Status 404 returned error can't find the container with id b623195e2605f7c1401a25807c925beedc104268409b494aae4efececd755e8c
Sep 17 18:19:41.366160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:41.366057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" event={"ID":"0d0dca22-7f6a-43bd-aa4f-600ad786b445","Type":"ContainerStarted","Data":"788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914"}
Sep 17 18:19:41.366160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:41.366122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" event={"ID":"0d0dca22-7f6a-43bd-aa4f-600ad786b445","Type":"ContainerStarted","Data":"b623195e2605f7c1401a25807c925beedc104268409b494aae4efececd755e8c"}
Sep 17 18:19:44.384535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:44.384502    2580 generic.go:356] "Generic (PLEG): container finished" podID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerID="788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914" exitCode=0
Sep 17 18:19:44.384913 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:44.384575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" event={"ID":"0d0dca22-7f6a-43bd-aa4f-600ad786b445","Type":"ContainerDied","Data":"788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914"}
Sep 17 18:19:45.391515 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:45.391482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" event={"ID":"0d0dca22-7f6a-43bd-aa4f-600ad786b445","Type":"ContainerStarted","Data":"7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b"}
Sep 17 18:19:45.414887 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:45.414835    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podStartSLOduration=2.75674895 podStartE2EDuration="5.414818313s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.658069363s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:45.412824865 +0000 UTC m=+2361.752218453" watchObservedRunningTime="2026-09-17 18:19:45.414818313 +0000 UTC m=+2361.754211905"
Sep 17 18:19:45.814535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:45.814456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:50.813807 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:50.813771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:19:50.815349 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:50.815319    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:19:54.280929 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:19:54.280890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:00.814187 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:00.814147    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:20:05.902772 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.902734    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"]
Sep 17 18:20:05.907249 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.907221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.910273 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.910250    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:20:05.910392 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.910337    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 18:20:05.910468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.910388    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 18:20:05.910468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.910285    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 18:20:05.910468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.910390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-xdp4t\""
Sep 17 18:20:05.910674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.910652    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 18:20:05.916711 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.916680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"]
Sep 17 18:20:05.929378 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.929349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:05.998182 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-dshm\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.998331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kserve-provision-location\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.998331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7rd\" (UniqueName: \"kubernetes.io/projected/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kube-api-access-tc7rd\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.998331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-model-cache\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.998331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tmp-dir\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.998494 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tls-certs\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.998494 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-home\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:05.998494 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:05.998446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-cabundle-cert\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099273 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-cabundle-cert\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099422 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-dshm\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099422 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kserve-provision-location\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099422 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7rd\" (UniqueName: \"kubernetes.io/projected/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kube-api-access-tc7rd\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099422 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-model-cache\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099422 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tmp-dir\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099422 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tls-certs\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-home\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tmp-dir\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.099899 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.099873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kserve-provision-location\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.100073 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.100052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-cabundle-cert\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.100073 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.100066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-home\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.100224 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.100187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-model-cache\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.101939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.101919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-dshm\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.102141 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.102126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tls-certs\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.109194 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.109173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7rd\" (UniqueName: \"kubernetes.io/projected/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kube-api-access-tc7rd\") pod \"router-with-refs-test-kserve-644444f686-qkk5n\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.188524 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.188446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:06.219104 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.219064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:06.279321 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.279289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:06.501803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:06.501775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"]
Sep 17 18:20:06.504217 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:20:06.504190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2124ef9_84e1_427e_a421_f2b7f4bfd92a.slice/crio-542b487893012d87e19595422a1b33509444d6343bb593613163029424fbd0c0 WatchSource:0}: Error finding container 542b487893012d87e19595422a1b33509444d6343bb593613163029424fbd0c0: Status 404 returned error can't find the container with id 542b487893012d87e19595422a1b33509444d6343bb593613163029424fbd0c0
Sep 17 18:20:07.513853 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:07.513811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" event={"ID":"f2124ef9-84e1-427e-a421-f2b7f4bfd92a","Type":"ContainerStarted","Data":"c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b"}
Sep 17 18:20:07.514233 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:07.513875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" event={"ID":"f2124ef9-84e1-427e-a421-f2b7f4bfd92a","Type":"ContainerStarted","Data":"542b487893012d87e19595422a1b33509444d6343bb593613163029424fbd0c0"}
Sep 17 18:20:09.526175 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:09.526137    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerID="c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b" exitCode=0
Sep 17 18:20:09.526648 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:09.526216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" event={"ID":"f2124ef9-84e1-427e-a421-f2b7f4bfd92a","Type":"ContainerDied","Data":"c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b"}
Sep 17 18:20:10.533507 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:10.533463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" event={"ID":"f2124ef9-84e1-427e-a421-f2b7f4bfd92a","Type":"ContainerStarted","Data":"bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469"}
Sep 17 18:20:10.556077 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:10.556017    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podStartSLOduration=3.347403075 podStartE2EDuration="5.555998132s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.208595057s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:10.554368576 +0000 UTC m=+2386.893762166" watchObservedRunningTime="2026-09-17 18:20:10.555998132 +0000 UTC m=+2386.895391722"
Sep 17 18:20:10.814508 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:10.814420    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:20:11.219187 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:11.219153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:16.219314 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:16.219280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:20:16.220754 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:16.220723    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:20:20.814120 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:20.814049    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:20:25.277193 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:25.277159    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:20:25.286345 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:25.286317    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:20:26.219495 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:26.219450    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:20:30.814727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:30.814671    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:20:36.219455 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:36.219413    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:20:40.615736 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:40.615699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:40.814939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:40.814892    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:20:46.219445 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:46.219356    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:20:50.814264 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:50.814223    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:20:52.012441 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:52.012404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:56.219434 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:20:56.219392    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:21:00.824136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:00.824105    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:21:00.832180 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:00.832155    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:21:02.617804 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:02.617760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:02.622585 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:02.622553    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"]
Sep 17 18:21:02.681200 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:02.681164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:06.220320 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:06.220280    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:21:07.723735 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:07.723701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:07.786908 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:07.786869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:16.219442 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:16.219402    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:21:17.648713 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:17.648651    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" containerID="cri-o://7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b" gracePeriod=45
Sep 17 18:21:17.825595 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:17.825556    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 18:21:18.394435 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.394409    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:21:18.537839 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.537760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-model-cache\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.537839 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.537825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kserve-provision-location\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.538040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.537867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-home\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.538040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.537889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tls-certs\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.538040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.537949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74rt5\" (UniqueName: \"kubernetes.io/projected/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kube-api-access-74rt5\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.538040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.537973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d0dca22-7f6a-43bd-aa4f-600ad786b445-cabundle-cert\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.538040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.538022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-dshm\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.538335 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.538044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tmp-dir\") pod \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\" (UID: \"0d0dca22-7f6a-43bd-aa4f-600ad786b445\") "
Sep 17 18:21:18.538335 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.538038    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:18.538335 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.538293    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.538488 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.538351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-home" (OuterVolumeSpecName: "home") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:18.538488 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.538418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0dca22-7f6a-43bd-aa4f-600ad786b445-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:18.538769 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.538741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:18.540356 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.540325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-dshm" (OuterVolumeSpecName: "dshm") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:18.540562 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.540530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:18.540669 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.540559    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kube-api-access-74rt5" (OuterVolumeSpecName: "kube-api-access-74rt5") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "kube-api-access-74rt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:18.561659 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.561635    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d0dca22-7f6a-43bd-aa4f-600ad786b445" (UID: "0d0dca22-7f6a-43bd-aa4f-600ad786b445"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:18.638866 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.638845    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74rt5\" (UniqueName: \"kubernetes.io/projected/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kube-api-access-74rt5\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.638866 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.638864    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d0dca22-7f6a-43bd-aa4f-600ad786b445-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.638995 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.638873    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.638995 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.638881    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.638995 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.638889    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.638995 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.638896    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0dca22-7f6a-43bd-aa4f-600ad786b445-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.638995 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.638904    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0dca22-7f6a-43bd-aa4f-600ad786b445-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:21:18.945710 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.945678    2580 generic.go:356] "Generic (PLEG): container finished" podID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerID="7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b" exitCode=0
Sep 17 18:21:18.946101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.945768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"
Sep 17 18:21:18.946101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.945766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" event={"ID":"0d0dca22-7f6a-43bd-aa4f-600ad786b445","Type":"ContainerDied","Data":"7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b"}
Sep 17 18:21:18.946101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.945918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh" event={"ID":"0d0dca22-7f6a-43bd-aa4f-600ad786b445","Type":"ContainerDied","Data":"b623195e2605f7c1401a25807c925beedc104268409b494aae4efececd755e8c"}
Sep 17 18:21:18.946101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.945942    2580 scope.go:117] "RemoveContainer" containerID="7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b"
Sep 17 18:21:18.955282 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.955262    2580 scope.go:117] "RemoveContainer" containerID="788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914"
Sep 17 18:21:18.964142 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.964124    2580 scope.go:117] "RemoveContainer" containerID="7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b"
Sep 17 18:21:18.964363 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:21:18.964347    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b\": container with ID starting with 7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b not found: ID does not exist" containerID="7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b"
Sep 17 18:21:18.964425 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.964370    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b"} err="failed to get container status \"7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b\": rpc error: code = NotFound desc = could not find container \"7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b\": container with ID starting with 7437d44a919beda89e9a6db722af597f62315fd371d28a9586ee14545838945b not found: ID does not exist"
Sep 17 18:21:18.964425 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.964386    2580 scope.go:117] "RemoveContainer" containerID="788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914"
Sep 17 18:21:18.964634 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:21:18.964620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914\": container with ID starting with 788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914 not found: ID does not exist" containerID="788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914"
Sep 17 18:21:18.964684 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.964635    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914"} err="failed to get container status \"788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914\": rpc error: code = NotFound desc = could not find container \"788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914\": container with ID starting with 788c1eaa60495a278a16d600a96beb230e6a5b501479c19c0dd1fedb0916e914 not found: ID does not exist"
Sep 17 18:21:18.973027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.973004    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"]
Sep 17 18:21:18.978236 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:18.978216    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c66df775d-flhqh"]
Sep 17 18:21:20.046010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:20.045981    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" path="/var/lib/kubelet/pods/0d0dca22-7f6a-43bd-aa4f-600ad786b445/volumes"
Sep 17 18:21:26.219623 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:26.219575    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:21:36.219535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:36.219482    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:21:46.219360 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:46.219321    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:21:52.103941 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:52.103908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:52.203832 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:52.203798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:56.228976 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:56.228945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:21:56.236543 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:56.236517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:21:58.399904 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:21:58.399865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:04.269501 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:04.269467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:04.369182 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:04.369150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:09.639273 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:09.639241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:09.924411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:09.924376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:27.523382 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:27.523296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:33.063490 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:33.063457    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"]
Sep 17 18:22:33.065469 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:33.065430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:33.125669 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:33.125638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:48.087414 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.087381    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" containerID="cri-o://bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469" gracePeriod=45
Sep 17 18:22:48.229635 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.229593    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 18:22:48.751684 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.751660    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:22:48.839231 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc7rd\" (UniqueName: \"kubernetes.io/projected/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kube-api-access-tc7rd\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839388 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tmp-dir\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839388 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-cabundle-cert\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839510 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kserve-provision-location\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839510 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tls-certs\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839510 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-dshm\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839658 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-home\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839658 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839556    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-model-cache\") pod \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\" (UID: \"f2124ef9-84e1-427e-a421-f2b7f4bfd92a\") "
Sep 17 18:22:48.839804 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:48.839983 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.839964    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:48.840594 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.840545    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-home" (OuterVolumeSpecName: "home") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:48.841760 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.841729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:48.841867 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.841840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-dshm" (OuterVolumeSpecName: "dshm") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:48.841934 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.841865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kube-api-access-tc7rd" (OuterVolumeSpecName: "kube-api-access-tc7rd") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "kube-api-access-tc7rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:48.859547 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.859517    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:48.868507 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.868484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2124ef9-84e1-427e-a421-f2b7f4bfd92a" (UID: "f2124ef9-84e1-427e-a421-f2b7f4bfd92a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:48.940780 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940761    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:48.940780 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940779    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:48.940900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940787    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:48.940900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940795    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:48.940900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940803    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc7rd\" (UniqueName: \"kubernetes.io/projected/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kube-api-access-tc7rd\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:48.940900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940811    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:48.940900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940819    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:48.940900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:48.940828    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2124ef9-84e1-427e-a421-f2b7f4bfd92a-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:22:49.433480 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.433452    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerID="bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469" exitCode=0
Sep 17 18:22:49.433878 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.433504    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"
Sep 17 18:22:49.433878 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.433540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" event={"ID":"f2124ef9-84e1-427e-a421-f2b7f4bfd92a","Type":"ContainerDied","Data":"bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469"}
Sep 17 18:22:49.433878 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.433597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n" event={"ID":"f2124ef9-84e1-427e-a421-f2b7f4bfd92a","Type":"ContainerDied","Data":"542b487893012d87e19595422a1b33509444d6343bb593613163029424fbd0c0"}
Sep 17 18:22:49.433878 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.433626    2580 scope.go:117] "RemoveContainer" containerID="bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469"
Sep 17 18:22:49.443031 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.443011    2580 scope.go:117] "RemoveContainer" containerID="c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b"
Sep 17 18:22:49.450636 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.450620    2580 scope.go:117] "RemoveContainer" containerID="bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469"
Sep 17 18:22:49.450895 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:22:49.450877    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469\": container with ID starting with bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469 not found: ID does not exist" containerID="bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469"
Sep 17 18:22:49.450934 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.450904    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469"} err="failed to get container status \"bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469\": rpc error: code = NotFound desc = could not find container \"bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469\": container with ID starting with bd28cf8f9cf15eb1ef4f71c11cb2460fb0f2e6e5a2bacb8c4a467d2d75989469 not found: ID does not exist"
Sep 17 18:22:49.450934 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.450920    2580 scope.go:117] "RemoveContainer" containerID="c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b"
Sep 17 18:22:49.451162 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:22:49.451139    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b\": container with ID starting with c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b not found: ID does not exist" containerID="c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b"
Sep 17 18:22:49.451204 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.451169    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b"} err="failed to get container status \"c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b\": rpc error: code = NotFound desc = could not find container \"c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b\": container with ID starting with c290e7c30e3099a9a96161f4e1dcc8530a15269939249d0ca27ef77333fef56b not found: ID does not exist"
Sep 17 18:22:49.456461 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.456440    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"]
Sep 17 18:22:49.462457 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:49.462437    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-644444f686-qkk5n"]
Sep 17 18:22:50.051234 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:50.051201    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" path="/var/lib/kubelet/pods/f2124ef9-84e1-427e-a421-f2b7f4bfd92a/volumes"
Sep 17 18:22:50.222789 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:50.222731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:50.309077 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:22:50.308981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:23.155476 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:23.155434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:24.657785 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:24.657736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:28.226687 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:28.226640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:44.249891 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.249851    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"]
Sep 17 18:23:44.250605 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250581    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="storage-initializer"
Sep 17 18:23:44.250709 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250607    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="storage-initializer"
Sep 17 18:23:44.250709 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250671    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="storage-initializer"
Sep 17 18:23:44.250709 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250691    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="storage-initializer"
Sep 17 18:23:44.250863 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main"
Sep 17 18:23:44.250863 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250726    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main"
Sep 17 18:23:44.250863 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250739    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main"
Sep 17 18:23:44.250863 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250747    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main"
Sep 17 18:23:44.251051 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0dca22-7f6a-43bd-aa4f-600ad786b445" containerName="main"
Sep 17 18:23:44.251051 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.250959    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2124ef9-84e1-427e-a421-f2b7f4bfd92a" containerName="main"
Sep 17 18:23:44.255459 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.255437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.260483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.260461    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:23:44.260613 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.260534    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 18:23:44.260677 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.260640    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-mscnm\""
Sep 17 18:23:44.260677 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.260670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-4wflw\""
Sep 17 18:23:44.260787 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.260755    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 18:23:44.260899 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.260881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 18:23:44.261247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.261227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 18:23:44.269883 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.269861    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"]
Sep 17 18:23:44.278721 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.278686    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"]
Sep 17 18:23:44.282295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.282275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.297756 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.297700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"]
Sep 17 18:23:44.319622 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.319593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:44.352146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dr28\" (UniqueName: \"kubernetes.io/projected/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kube-api-access-2dr28\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6da41c-5c42-4245-9944-5c97dc2c9c89-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.352368 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjgs\" (UniqueName: \"kubernetes.io/projected/d3219b30-323f-485f-9445-c947e3928613-kube-api-access-kfjgs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.352368 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352368 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352464 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.352464 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.352464 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352464 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3219b30-323f-485f-9445-c947e3928613-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.352585 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352585 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.352585 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.352585 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.352707 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.352593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3219b30-323f-485f-9445-c947e3928613-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453227 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjgs\" (UniqueName: \"kubernetes.io/projected/d3219b30-323f-485f-9445-c947e3928613-kube-api-access-kfjgs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.453342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.453342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453520 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453520 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.453520 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3219b30-323f-485f-9445-c947e3928613-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453520 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3219b30-323f-485f-9445-c947e3928613-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.453727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dr28\" (UniqueName: \"kubernetes.io/projected/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kube-api-access-2dr28\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.454148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.454148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6da41c-5c42-4245-9944-5c97dc2c9c89-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.454148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.454148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.454148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.454148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.453984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.454148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.454010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.454514 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.454486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.454647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.454624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6da41c-5c42-4245-9944-5c97dc2c9c89-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.454903 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.454876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3219b30-323f-485f-9445-c947e3928613-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.456047 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.456026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.456192 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.456171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.456354 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.456337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.456560 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.456537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3219b30-323f-485f-9445-c947e3928613-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.461615 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.461589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjgs\" (UniqueName: \"kubernetes.io/projected/d3219b30-323f-485f-9445-c947e3928613-kube-api-access-kfjgs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.461707 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.461650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dr28\" (UniqueName: \"kubernetes.io/projected/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kube-api-access-2dr28\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.566336 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.566280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:44.594574 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.594548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:44.847777 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.847752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"]
Sep 17 18:23:44.848834 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:23:44.848811    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3219b30_323f_485f_9445_c947e3928613.slice/crio-bbc5b9d8912c228ab7e9a6bd201d37a82cf3d2d9e0ba99f7c811662f88a0cc68 WatchSource:0}: Error finding container bbc5b9d8912c228ab7e9a6bd201d37a82cf3d2d9e0ba99f7c811662f88a0cc68: Status 404 returned error can't find the container with id bbc5b9d8912c228ab7e9a6bd201d37a82cf3d2d9e0ba99f7c811662f88a0cc68
Sep 17 18:23:44.868079 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.868051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:44.879411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:44.879388    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"]
Sep 17 18:23:44.884150 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:23:44.884123    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6da41c_5c42_4245_9944_5c97dc2c9c89.slice/crio-1abb37766941d31f0aa791c4b1f701fd572456624cc8025fc29699cac37c881b WatchSource:0}: Error finding container 1abb37766941d31f0aa791c4b1f701fd572456624cc8025fc29699cac37c881b: Status 404 returned error can't find the container with id 1abb37766941d31f0aa791c4b1f701fd572456624cc8025fc29699cac37c881b
Sep 17 18:23:45.732871 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:45.732827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" event={"ID":"8c6da41c-5c42-4245-9944-5c97dc2c9c89","Type":"ContainerStarted","Data":"742b4031a5d0b0a2885a3a89a884f64f67b29d26830db7996a558cdd253662ce"}
Sep 17 18:23:45.733355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:45.732904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" event={"ID":"8c6da41c-5c42-4245-9944-5c97dc2c9c89","Type":"ContainerStarted","Data":"1abb37766941d31f0aa791c4b1f701fd572456624cc8025fc29699cac37c881b"}
Sep 17 18:23:45.735160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:45.735113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerStarted","Data":"26a353316a760917096dbc1981af21d8e9cd861cb16fbffc3b3398e12f5f1205"}
Sep 17 18:23:45.735316 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:45.735194    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:45.735316 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:45.735265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerStarted","Data":"bbc5b9d8912c228ab7e9a6bd201d37a82cf3d2d9e0ba99f7c811662f88a0cc68"}
Sep 17 18:23:46.748136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:46.748077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerStarted","Data":"b721e603853b2291b3c0d02def5b5aac81a6c948f3c1f1f5f62cedc5246a44a6"}
Sep 17 18:23:47.760497 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:47.760456    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerID="742b4031a5d0b0a2885a3a89a884f64f67b29d26830db7996a558cdd253662ce" exitCode=0
Sep 17 18:23:47.762255 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:47.760503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" event={"ID":"8c6da41c-5c42-4245-9944-5c97dc2c9c89","Type":"ContainerDied","Data":"742b4031a5d0b0a2885a3a89a884f64f67b29d26830db7996a558cdd253662ce"}
Sep 17 18:23:48.770490 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:48.770449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" event={"ID":"8c6da41c-5c42-4245-9944-5c97dc2c9c89","Type":"ContainerStarted","Data":"6ba6b6c19966ea1ca3344662b6858ac6e063226f8489bd002181867ed424ad0a"}
Sep 17 18:23:48.772544 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:48.772517    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3219b30-323f-485f-9445-c947e3928613" containerID="b721e603853b2291b3c0d02def5b5aac81a6c948f3c1f1f5f62cedc5246a44a6" exitCode=0
Sep 17 18:23:48.772656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:48.772587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerDied","Data":"b721e603853b2291b3c0d02def5b5aac81a6c948f3c1f1f5f62cedc5246a44a6"}
Sep 17 18:23:48.796473 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:48.796420    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podStartSLOduration=2.511410403 podStartE2EDuration="4.796401117s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.284990714s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:48.791695356 +0000 UTC m=+2605.131088950" watchObservedRunningTime="2026-09-17 18:23:48.796401117 +0000 UTC m=+2605.135794709"
Sep 17 18:23:49.321151 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:49.321113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:49.782186 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:49.782147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerStarted","Data":"d9c4f05cc5ad302f31dba10a4dc5f0c087e14a0cb47e21fbb66e34978bb98e09"}
Sep 17 18:23:49.813007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:49.812959    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podStartSLOduration=3.393497809 podStartE2EDuration="5.812943603s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.419445794s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:49.810152925 +0000 UTC m=+2606.149546506" watchObservedRunningTime="2026-09-17 18:23:49.812943603 +0000 UTC m=+2606.152337193"
Sep 17 18:23:50.738903 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:50.738861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:54.567252 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:54.567209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:54.567685 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:54.567291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:54.568576 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:54.568525    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:23:54.585606 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:54.585581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:23:54.595410 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:54.595382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:54.595484 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:54.595451    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:23:54.596720 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:54.596679    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:23:56.606871 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:23:56.606815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:04.566883 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:04.566840    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:24:04.595895 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:04.595857    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:24:07.507242 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:07.507205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:12.719702 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:12.719665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:12.803794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:12.803757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:12.904098 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:12.904040    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:14.567300 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:14.567257    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:24:14.595788 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:14.595744    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:24:15.724993 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:15.724953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:20.125122 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:20.125064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:20.188157 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:20.188114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:24.567370 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:24.567318    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:24:24.596017 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:24.595974    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:24:26.419819 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:26.419780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:26.589079 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:26.589022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:34.567031 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:34.566984    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:24:34.595979 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:34.595942    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:24:44.567221 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:44.567162    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:24:44.595727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:44.595683    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:24:44.933872 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:44.933829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:48.214932 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:48.214890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:54.567598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:54.567544    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:24:54.595996 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:24:54.595949    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:25:04.567649 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:04.567589    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:25:04.595340 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:04.595297    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:25:09.607368 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:09.607326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:14.567067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:14.566973    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:25:14.595549 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:14.595513    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:25:17.219830 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:17.219786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:24.567515 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:24.567468    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:25:24.595899 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:24.595864    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:25:25.318532 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:25.318506    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:25:25.327336 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:25.327316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:25:34.567447 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:34.567398    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:25:34.595171 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:34.595133    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:25:44.567295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:44.567244    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:25:44.595610 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:44.595571    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:25:54.567623 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:54.567573    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:25:54.594962 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:25:54.594916    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:26:04.567247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:04.567197    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 17 18:26:04.595749 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:04.595707    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 18:26:14.576189 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:14.576159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:26:14.605966 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:14.605938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:26:14.613379 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:14.613348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:26:14.652123 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:14.652080    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:26:22.833354 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:22.833315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:22.849710 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:22.849678    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"]
Sep 17 18:26:22.853120 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:22.853076    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"]
Sep 17 18:26:22.899683 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:22.899648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:28.036248 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:28.036213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:28.129182 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:28.129131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:34.772595 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:34.772564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:37.874935 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:37.874894    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main" containerID="cri-o://d9c4f05cc5ad302f31dba10a4dc5f0c087e14a0cb47e21fbb66e34978bb98e09" gracePeriod=45
Sep 17 18:26:37.875357 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:37.874973    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main" containerID="cri-o://6ba6b6c19966ea1ca3344662b6858ac6e063226f8489bd002181867ed424ad0a" gracePeriod=45
Sep 17 18:26:38.625916 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.625869    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="llm-d-routing-sidecar" containerID="cri-o://26a353316a760917096dbc1981af21d8e9cd861cb16fbffc3b3398e12f5f1205" gracePeriod=45
Sep 17 18:26:38.771232 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.771192    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerID="6ba6b6c19966ea1ca3344662b6858ac6e063226f8489bd002181867ed424ad0a" exitCode=0
Sep 17 18:26:38.771415 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.771280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" event={"ID":"8c6da41c-5c42-4245-9944-5c97dc2c9c89","Type":"ContainerDied","Data":"6ba6b6c19966ea1ca3344662b6858ac6e063226f8489bd002181867ed424ad0a"}
Sep 17 18:26:38.774027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.773843    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3219b30-323f-485f-9445-c947e3928613" containerID="d9c4f05cc5ad302f31dba10a4dc5f0c087e14a0cb47e21fbb66e34978bb98e09" exitCode=0
Sep 17 18:26:38.774027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.773868    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3219b30-323f-485f-9445-c947e3928613" containerID="26a353316a760917096dbc1981af21d8e9cd861cb16fbffc3b3398e12f5f1205" exitCode=0
Sep 17 18:26:38.774027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.773949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerDied","Data":"d9c4f05cc5ad302f31dba10a4dc5f0c087e14a0cb47e21fbb66e34978bb98e09"}
Sep 17 18:26:38.774027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.773990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerDied","Data":"26a353316a760917096dbc1981af21d8e9cd861cb16fbffc3b3398e12f5f1205"}
Sep 17 18:26:38.890198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.890174    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:26:38.892974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.892956    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:26:38.973294 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tmp-dir\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-tmp-dir\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.973432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-model-cache\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3219b30-323f-485f-9445-c947e3928613-cabundle-cert\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.973432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-dshm\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-kserve-provision-location\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.973432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3219b30-323f-485f-9445-c947e3928613-tls-certs\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-home\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-model-cache\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tls-certs\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973545    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dr28\" (UniqueName: \"kubernetes.io/projected/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kube-api-access-2dr28\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kserve-provision-location\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-home\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6da41c-5c42-4245-9944-5c97dc2c9c89-cabundle-cert\") pod \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\" (UID: \"8c6da41c-5c42-4245-9944-5c97dc2c9c89\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973635    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973669    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-dshm\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.973739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfjgs\" (UniqueName: \"kubernetes.io/projected/d3219b30-323f-485f-9445-c947e3928613-kube-api-access-kfjgs\") pod \"d3219b30-323f-485f-9445-c947e3928613\" (UID: \"d3219b30-323f-485f-9445-c947e3928613\") "
Sep 17 18:26:38.974266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3219b30-323f-485f-9445-c947e3928613-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:38.974266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.973912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.975758 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.974081    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:38.975758 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.975393    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3219b30-323f-485f-9445-c947e3928613-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:38.975758 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.975409    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:38.975758 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.975357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-home" (OuterVolumeSpecName: "home") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.976035 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.975926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6da41c-5c42-4245-9944-5c97dc2c9c89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:38.976172 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.976133    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-dshm" (OuterVolumeSpecName: "dshm") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.976498 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.976470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-home" (OuterVolumeSpecName: "home") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.976968 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.976946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-dshm" (OuterVolumeSpecName: "dshm") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.977062 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.977034    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3219b30-323f-485f-9445-c947e3928613-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:38.977417 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.977390    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3219b30-323f-485f-9445-c947e3928613-kube-api-access-kfjgs" (OuterVolumeSpecName: "kube-api-access-kfjgs") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "kube-api-access-kfjgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:38.978037 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.978014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:38.978370 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.978345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kube-api-access-2dr28" (OuterVolumeSpecName: "kube-api-access-2dr28") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "kube-api-access-2dr28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:38.984522 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.984493    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.989140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.989120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3219b30-323f-485f-9445-c947e3928613" (UID: "d3219b30-323f-485f-9445-c947e3928613"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:38.989283 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:38.989262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:39.000610 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.000586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c6da41c-5c42-4245-9944-5c97dc2c9c89" (UID: "8c6da41c-5c42-4245-9944-5c97dc2c9c89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:39.076516 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076495    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dr28\" (UniqueName: \"kubernetes.io/projected/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kube-api-access-2dr28\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076524    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076534    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076542    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6da41c-5c42-4245-9944-5c97dc2c9c89-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076550    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076558    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjgs\" (UniqueName: \"kubernetes.io/projected/d3219b30-323f-485f-9445-c947e3928613-kube-api-access-kfjgs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076565    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076573    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076579    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076586    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3219b30-323f-485f-9445-c947e3928613-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076594    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3219b30-323f-485f-9445-c947e3928613-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076601    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6da41c-5c42-4245-9944-5c97dc2c9c89-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.076896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.076609    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6da41c-5c42-4245-9944-5c97dc2c9c89-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:26:39.780664 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.780638    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"
Sep 17 18:26:39.780845 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.780673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778" event={"ID":"d3219b30-323f-485f-9445-c947e3928613","Type":"ContainerDied","Data":"bbc5b9d8912c228ab7e9a6bd201d37a82cf3d2d9e0ba99f7c811662f88a0cc68"}
Sep 17 18:26:39.780845 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.780734    2580 scope.go:117] "RemoveContainer" containerID="d9c4f05cc5ad302f31dba10a4dc5f0c087e14a0cb47e21fbb66e34978bb98e09"
Sep 17 18:26:39.784695 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.784666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch" event={"ID":"8c6da41c-5c42-4245-9944-5c97dc2c9c89","Type":"ContainerDied","Data":"1abb37766941d31f0aa791c4b1f701fd572456624cc8025fc29699cac37c881b"}
Sep 17 18:26:39.784887 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.784869    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"
Sep 17 18:26:39.797856 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.797569    2580 scope.go:117] "RemoveContainer" containerID="b721e603853b2291b3c0d02def5b5aac81a6c948f3c1f1f5f62cedc5246a44a6"
Sep 17 18:26:39.807406 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.807388    2580 scope.go:117] "RemoveContainer" containerID="26a353316a760917096dbc1981af21d8e9cd861cb16fbffc3b3398e12f5f1205"
Sep 17 18:26:39.815606 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.815585    2580 scope.go:117] "RemoveContainer" containerID="6ba6b6c19966ea1ca3344662b6858ac6e063226f8489bd002181867ed424ad0a"
Sep 17 18:26:39.816450 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.816426    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"]
Sep 17 18:26:39.820518 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.820491    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-77m84ch"]
Sep 17 18:26:39.825450 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.825433    2580 scope.go:117] "RemoveContainer" containerID="742b4031a5d0b0a2885a3a89a884f64f67b29d26830db7996a558cdd253662ce"
Sep 17 18:26:39.830823 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.830796    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"]
Sep 17 18:26:39.837265 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:39.837241    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8657b5dd989c778"]
Sep 17 18:26:40.045565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:40.045539    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" path="/var/lib/kubelet/pods/8c6da41c-5c42-4245-9944-5c97dc2c9c89/volumes"
Sep 17 18:26:40.045985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:26:40.045972    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3219b30-323f-485f-9445-c947e3928613" path="/var/lib/kubelet/pods/d3219b30-323f-485f-9445-c947e3928613/volumes"
Sep 17 18:27:12.944990 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:12.944951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:13.043581 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:13.043551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:13.147040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:13.147004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:17.807281 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:17.807244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:17.901666 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:17.901636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:19.670310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:19.670258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:23.958063 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:23.958029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:25.378979 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:25.378948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:25.523916 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:25.523854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:28.143069 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143037    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"]
Sep 17 18:27:28.143596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143574    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="storage-initializer"
Sep 17 18:27:28.143681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143599    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="storage-initializer"
Sep 17 18:27:28.143681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143622    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main"
Sep 17 18:27:28.143681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143630    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main"
Sep 17 18:27:28.143681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143641    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="llm-d-routing-sidecar"
Sep 17 18:27:28.143681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143650    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="llm-d-routing-sidecar"
Sep 17 18:27:28.143681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143669    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="storage-initializer"
Sep 17 18:27:28.143681 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143677    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="storage-initializer"
Sep 17 18:27:28.144033 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143691    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main"
Sep 17 18:27:28.144033 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143699    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main"
Sep 17 18:27:28.144033 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143787    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="llm-d-routing-sidecar"
Sep 17 18:27:28.144033 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143806    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6da41c-5c42-4245-9944-5c97dc2c9c89" containerName="main"
Sep 17 18:27:28.144033 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.143815    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3219b30-323f-485f-9445-c947e3928613" containerName="main"
Sep 17 18:27:28.148745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.148724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.151367 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.151348    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:27:28.151536 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.151504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-bb6zz\""
Sep 17 18:27:28.151789 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.151774    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 18:27:28.151858 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.151805    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:27:28.151909 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.151853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:27:28.151954 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.151932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-l2978\""
Sep 17 18:27:28.152046 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.152029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:27:28.162276 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.162246    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"]
Sep 17 18:27:28.207369 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.207333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:28.238451 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-dshm\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.238451 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81402b7c-cd33-4055-b5ad-158e244ae347-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.238646 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81402b7c-cd33-4055-b5ad-158e244ae347-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.238646 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.238646 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.238646 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq54v\" (UniqueName: \"kubernetes.io/projected/81402b7c-cd33-4055-b5ad-158e244ae347-kube-api-access-qq54v\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.238891 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.238891 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.238694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-home\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339454 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339569 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq54v\" (UniqueName: \"kubernetes.io/projected/81402b7c-cd33-4055-b5ad-158e244ae347-kube-api-access-qq54v\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339569 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339569 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-home\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339569 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-dshm\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81402b7c-cd33-4055-b5ad-158e244ae347-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81402b7c-cd33-4055-b5ad-158e244ae347-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339889 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339925 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.339970 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-home\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.340006 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.339974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.340320 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.340302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81402b7c-cd33-4055-b5ad-158e244ae347-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.341712 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.341693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-dshm\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.342129 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.342112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81402b7c-cd33-4055-b5ad-158e244ae347-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.347934 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.347913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq54v\" (UniqueName: \"kubernetes.io/projected/81402b7c-cd33-4055-b5ad-158e244ae347-kube-api-access-qq54v\") pod \"custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.459739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.459669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:28.461019 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.460992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:28.740344 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:28.740316    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"]
Sep 17 18:27:28.741230 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:27:28.741205    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81402b7c_cd33_4055_b5ad_158e244ae347.slice/crio-9d92fb7ee4d500ec39dc75c752dba935af32fdc060771269ef8b75e6a0fa283f WatchSource:0}: Error finding container 9d92fb7ee4d500ec39dc75c752dba935af32fdc060771269ef8b75e6a0fa283f: Status 404 returned error can't find the container with id 9d92fb7ee4d500ec39dc75c752dba935af32fdc060771269ef8b75e6a0fa283f
Sep 17 18:27:29.041514 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:29.041435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerStarted","Data":"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"}
Sep 17 18:27:29.041514 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:29.041477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerStarted","Data":"9d92fb7ee4d500ec39dc75c752dba935af32fdc060771269ef8b75e6a0fa283f"}
Sep 17 18:27:29.041690 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:29.041568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:30.050640 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:30.050606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerStarted","Data":"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"}
Sep 17 18:27:30.182703 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:30.182657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:30.283565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:30.283531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:32.064465 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:32.064429    2580 generic.go:356] "Generic (PLEG): container finished" podID="81402b7c-cd33-4055-b5ad-158e244ae347" containerID="f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73" exitCode=0
Sep 17 18:27:32.064900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:32.064491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerDied","Data":"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"}
Sep 17 18:27:33.073171 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:33.073137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerStarted","Data":"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"}
Sep 17 18:27:33.097404 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:33.097352    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podStartSLOduration=2.966518914 podStartE2EDuration="5.09733667s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.130817756s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:33.095686645 +0000 UTC m=+2829.435080238" watchObservedRunningTime="2026-09-17 18:27:33.09733667 +0000 UTC m=+2829.436730259"
Sep 17 18:27:37.409519 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:37.409470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:38.460144 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:38.460104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:38.460564 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:38.460190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:38.461657 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:38.461618    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:27:38.473603 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:38.473582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:27:48.460358 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:48.460309    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:27:58.461033 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:27:58.460989    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:28:00.403302 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.403262    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"]
Sep 17 18:28:00.408060 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.408040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.410822 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.410798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 18:28:00.410952 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.410867    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:28:00.410952 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.410831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 18:28:00.410952 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.410884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-qx57v\""
Sep 17 18:28:00.411154 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.410842    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 18:28:00.411154 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.411012    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:28:00.419449 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.419425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"]
Sep 17 18:28:00.477662 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.477625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:00.501256 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-dshm\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.501387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/907bed2a-da20-4a04-bc5a-311d7a5f45a8-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.501387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tls-certs\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.501387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4sb7\" (UniqueName: \"kubernetes.io/projected/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kube-api-access-x4sb7\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.501507 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tmp-dir\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.501507 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-home\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.501507 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.501633 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.501519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-model-cache\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.602414 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.602370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4sb7\" (UniqueName: \"kubernetes.io/projected/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kube-api-access-x4sb7\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.602733 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.602708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tmp-dir\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.602906 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.602889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-home\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.603031 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.603184 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-model-cache\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.603319 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tmp-dir\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.603420 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-dshm\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.603527 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/907bed2a-da20-4a04-bc5a-311d7a5f45a8-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.603641 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tls-certs\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.603841 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-model-cache\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.604121 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.604098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.604540 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.603579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-home\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.604861 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.604834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/907bed2a-da20-4a04-bc5a-311d7a5f45a8-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.607802 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.607778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tls-certs\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.609533 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.609504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-dshm\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.613913 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.613886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4sb7\" (UniqueName: \"kubernetes.io/projected/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kube-api-access-x4sb7\") pod \"lora-single-adapter-test-kserve-8d64fdb-lq2zn\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:00.719356 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:00.719293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:01.004920 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:01.004889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"]
Sep 17 18:28:01.006182 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:28:01.006150    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907bed2a_da20_4a04_bc5a_311d7a5f45a8.slice/crio-86910bd0be4073c03af76030dbf3f80e8dd3d29d708a08a722ffbb7afb1b451a WatchSource:0}: Error finding container 86910bd0be4073c03af76030dbf3f80e8dd3d29d708a08a722ffbb7afb1b451a: Status 404 returned error can't find the container with id 86910bd0be4073c03af76030dbf3f80e8dd3d29d708a08a722ffbb7afb1b451a
Sep 17 18:28:01.228153 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:01.228112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" event={"ID":"907bed2a-da20-4a04-bc5a-311d7a5f45a8","Type":"ContainerStarted","Data":"612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487"}
Sep 17 18:28:01.228342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:01.228183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" event={"ID":"907bed2a-da20-4a04-bc5a-311d7a5f45a8","Type":"ContainerStarted","Data":"86910bd0be4073c03af76030dbf3f80e8dd3d29d708a08a722ffbb7afb1b451a"}
Sep 17 18:28:04.251722 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:04.251682    2580 generic.go:356] "Generic (PLEG): container finished" podID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerID="612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487" exitCode=0
Sep 17 18:28:04.252104 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:04.251757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" event={"ID":"907bed2a-da20-4a04-bc5a-311d7a5f45a8","Type":"ContainerDied","Data":"612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487"}
Sep 17 18:28:05.259975 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:05.259937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" event={"ID":"907bed2a-da20-4a04-bc5a-311d7a5f45a8","Type":"ContainerStarted","Data":"c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89"}
Sep 17 18:28:05.283094 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:05.283023    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podStartSLOduration=2.5294365770000002 podStartE2EDuration="5.28300154s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.753564963s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:05.281622183 +0000 UTC m=+2861.621015783" watchObservedRunningTime="2026-09-17 18:28:05.28300154 +0000 UTC m=+2861.622395130"
Sep 17 18:28:05.720341 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:05.720305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:08.460763 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:08.460708    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:28:10.720329 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:10.720290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:28:10.721939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:10.721901    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:28:18.461168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:18.461046    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:28:20.719680 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:20.719639    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:28:28.461101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:28.461047    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:28:30.720519 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:30.720465    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:28:38.460570 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:38.460515    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:28:40.720282 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:40.720234    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:28:48.460973 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:48.460921    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:28:50.720302 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:50.720248    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:28:58.461026 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:28:58.460963    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:29:00.720261 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:00.720210    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:29:08.460771 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:08.460705    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:29:10.719611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:10.719572    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:29:18.460158 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:18.460111    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8001/health\": dial tcp 10.132.0.75:8001: connect: connection refused"
Sep 17 18:29:20.729867 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:20.729836    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:29:20.737225 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:20.737196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:29:28.469773 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:28.469741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:29:28.487562 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:28.487536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:29:38.920877 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:38.920828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:39.665402 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:39.665363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"]
Sep 17 18:29:40.172488 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:40.172452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:47.188160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:47.188068    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-8d64fdb-lq2zn_907bed2a-da20-4a04-bc5a-311d7a5f45a8/storage-initializer/0.log"
Sep 17 18:29:47.199368 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:47.199343    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-8d64fdb-lq2zn_907bed2a-da20-4a04-bc5a-311d7a5f45a8/main/0.log"
Sep 17 18:29:48.478152 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:48.478114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"]
Sep 17 18:29:48.504148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:48.504110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:49.662687 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:49.662643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.690430 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:54.690372    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main" containerID="cri-o://44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708" gracePeriod=45
Sep 17 18:29:55.308679 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.308644    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="llm-d-routing-sidecar" containerID="cri-o://ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2" gracePeriod=45
Sep 17 18:29:55.555383 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.555358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:29:55.709356 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-model-cache\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-dshm\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-tmp-dir\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq54v\" (UniqueName: \"kubernetes.io/projected/81402b7c-cd33-4055-b5ad-158e244ae347-kube-api-access-qq54v\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-home\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-kserve-provision-location\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81402b7c-cd33-4055-b5ad-158e244ae347-cabundle-cert\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81402b7c-cd33-4055-b5ad-158e244ae347-tls-certs\") pod \"81402b7c-cd33-4055-b5ad-158e244ae347\" (UID: \"81402b7c-cd33-4055-b5ad-158e244ae347\") "
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-model-cache" (OuterVolumeSpecName: "model-cache") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.709794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709788    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.710309 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.709976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81402b7c-cd33-4055-b5ad-158e244ae347-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:55.710309 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.710232    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-home" (OuterVolumeSpecName: "home") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.711907 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.711879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-dshm" (OuterVolumeSpecName: "dshm") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.712047 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.711887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81402b7c-cd33-4055-b5ad-158e244ae347-kube-api-access-qq54v" (OuterVolumeSpecName: "kube-api-access-qq54v") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "kube-api-access-qq54v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:55.712047 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.711983    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81402b7c-cd33-4055-b5ad-158e244ae347-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:55.721377 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.721348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.721878 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.721855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81402b7c-cd33-4055-b5ad-158e244ae347" (UID: "81402b7c-cd33-4055-b5ad-158e244ae347"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:55.810160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.810109    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.810160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.810129    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.810160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.810137    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq54v\" (UniqueName: \"kubernetes.io/projected/81402b7c-cd33-4055-b5ad-158e244ae347-kube-api-access-qq54v\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.810160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.810147    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.810160 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.810156    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81402b7c-cd33-4055-b5ad-158e244ae347-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.810375 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.810166    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81402b7c-cd33-4055-b5ad-158e244ae347-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.810375 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.810174    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81402b7c-cd33-4055-b5ad-158e244ae347-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.928573 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.928541    2580 generic.go:356] "Generic (PLEG): container finished" podID="81402b7c-cd33-4055-b5ad-158e244ae347" containerID="44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708" exitCode=0
Sep 17 18:29:55.928573 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.928565    2580 generic.go:356] "Generic (PLEG): container finished" podID="81402b7c-cd33-4055-b5ad-158e244ae347" containerID="ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2" exitCode=0
Sep 17 18:29:55.928756 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.928589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerDied","Data":"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"}
Sep 17 18:29:55.928756 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.928597    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"
Sep 17 18:29:55.928756 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.928674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerDied","Data":"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"}
Sep 17 18:29:55.928756 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.928702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv" event={"ID":"81402b7c-cd33-4055-b5ad-158e244ae347","Type":"ContainerDied","Data":"9d92fb7ee4d500ec39dc75c752dba935af32fdc060771269ef8b75e6a0fa283f"}
Sep 17 18:29:55.928756 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.928732    2580 scope.go:117] "RemoveContainer" containerID="44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"
Sep 17 18:29:55.937600 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.937581    2580 scope.go:117] "RemoveContainer" containerID="f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"
Sep 17 18:29:55.945918 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.945901    2580 scope.go:117] "RemoveContainer" containerID="ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"
Sep 17 18:29:55.953828 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.953809    2580 scope.go:117] "RemoveContainer" containerID="44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"
Sep 17 18:29:55.953882 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.953856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"]
Sep 17 18:29:55.954112 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:29:55.954074    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708\": container with ID starting with 44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708 not found: ID does not exist" containerID="44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"
Sep 17 18:29:55.954157 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954121    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"} err="failed to get container status \"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708\": rpc error: code = NotFound desc = could not find container \"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708\": container with ID starting with 44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708 not found: ID does not exist"
Sep 17 18:29:55.954157 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954139    2580 scope.go:117] "RemoveContainer" containerID="f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"
Sep 17 18:29:55.954404 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:29:55.954384    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73\": container with ID starting with f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73 not found: ID does not exist" containerID="f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"
Sep 17 18:29:55.954442 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954410    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"} err="failed to get container status \"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73\": rpc error: code = NotFound desc = could not find container \"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73\": container with ID starting with f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73 not found: ID does not exist"
Sep 17 18:29:55.954442 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954425    2580 scope.go:117] "RemoveContainer" containerID="ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"
Sep 17 18:29:55.954658 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:29:55.954637    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2\": container with ID starting with ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2 not found: ID does not exist" containerID="ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"
Sep 17 18:29:55.954706 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954663    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"} err="failed to get container status \"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2\": rpc error: code = NotFound desc = could not find container \"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2\": container with ID starting with ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2 not found: ID does not exist"
Sep 17 18:29:55.954706 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954684    2580 scope.go:117] "RemoveContainer" containerID="44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"
Sep 17 18:29:55.954913 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954894    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708"} err="failed to get container status \"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708\": rpc error: code = NotFound desc = could not find container \"44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708\": container with ID starting with 44208e38c737dfec4a98659f7fb156bcf8270cf038da1acfb2267f565e363708 not found: ID does not exist"
Sep 17 18:29:55.954956 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.954913    2580 scope.go:117] "RemoveContainer" containerID="f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"
Sep 17 18:29:55.955156 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.955131    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73"} err="failed to get container status \"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73\": rpc error: code = NotFound desc = could not find container \"f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73\": container with ID starting with f52434bdeb14e49dc44448660329802159cd5ba9f0fe9c72ea8f3dd4244a1c73 not found: ID does not exist"
Sep 17 18:29:55.955156 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.955150    2580 scope.go:117] "RemoveContainer" containerID="ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"
Sep 17 18:29:55.955384 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.955359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2"} err="failed to get container status \"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2\": rpc error: code = NotFound desc = could not find container \"ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2\": container with ID starting with ee2b9d250fd67146b711491723f9b17b517bc88576deb4f7b70aceccf01090b2 not found: ID does not exist"
Sep 17 18:29:55.958456 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:55.958433    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5596fb86c5-9pvjv"]
Sep 17 18:29:56.046443 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:29:56.046418    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" path="/var/lib/kubelet/pods/81402b7c-cd33-4055-b5ad-158e244ae347/volumes"
Sep 17 18:30:03.504562 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:03.504504    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" containerID="cri-o://c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89" gracePeriod=45
Sep 17 18:30:03.730949 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:03.730900    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 18:30:04.159577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.159549    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:30:04.271709 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-home\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.271709 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-dshm\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.271939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tmp-dir\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.271939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tls-certs\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.271939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kserve-provision-location\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.271939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-model-cache\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.271939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4sb7\" (UniqueName: \"kubernetes.io/projected/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kube-api-access-x4sb7\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.271939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.271897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/907bed2a-da20-4a04-bc5a-311d7a5f45a8-cabundle-cert\") pod \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\" (UID: \"907bed2a-da20-4a04-bc5a-311d7a5f45a8\") "
Sep 17 18:30:04.272248 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.272151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-home" (OuterVolumeSpecName: "home") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:04.272456 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.272382    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:04.272591 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.272478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907bed2a-da20-4a04-bc5a-311d7a5f45a8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:04.274110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.274070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-dshm" (OuterVolumeSpecName: "dshm") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:04.274211 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.274123    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kube-api-access-x4sb7" (OuterVolumeSpecName: "kube-api-access-x4sb7") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "kube-api-access-x4sb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:04.274256 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.274245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:04.286187 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.286160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:04.297301 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.297281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "907bed2a-da20-4a04-bc5a-311d7a5f45a8" (UID: "907bed2a-da20-4a04-bc5a-311d7a5f45a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:04.372944 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372920    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4sb7\" (UniqueName: \"kubernetes.io/projected/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kube-api-access-x4sb7\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.372944 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372942    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/907bed2a-da20-4a04-bc5a-311d7a5f45a8-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.373074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372951    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.373074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372959    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.373074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372967    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.373074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372974    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907bed2a-da20-4a04-bc5a-311d7a5f45a8-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.373074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372982    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.373074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.372989    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/907bed2a-da20-4a04-bc5a-311d7a5f45a8-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:30:04.834580 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.834554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.978136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.978111    2580 generic.go:356] "Generic (PLEG): container finished" podID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerID="c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89" exitCode=0
Sep 17 18:30:04.978250 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.978158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" event={"ID":"907bed2a-da20-4a04-bc5a-311d7a5f45a8","Type":"ContainerDied","Data":"c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89"}
Sep 17 18:30:04.978250 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.978196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn" event={"ID":"907bed2a-da20-4a04-bc5a-311d7a5f45a8","Type":"ContainerDied","Data":"86910bd0be4073c03af76030dbf3f80e8dd3d29d708a08a722ffbb7afb1b451a"}
Sep 17 18:30:04.978250 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.978218    2580 scope.go:117] "RemoveContainer" containerID="c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89"
Sep 17 18:30:04.978250 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.978159    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"
Sep 17 18:30:04.988238 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.988220    2580 scope.go:117] "RemoveContainer" containerID="612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487"
Sep 17 18:30:04.999687 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:04.999666    2580 scope.go:117] "RemoveContainer" containerID="c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89"
Sep 17 18:30:05.000005 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:30:04.999967    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89\": container with ID starting with c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89 not found: ID does not exist" containerID="c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89"
Sep 17 18:30:05.000110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:05.000012    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89"} err="failed to get container status \"c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89\": rpc error: code = NotFound desc = could not find container \"c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89\": container with ID starting with c8af55f1c5238fada91b178f0cb1a453c5c5bd96471eacc03a5a1186abfa8c89 not found: ID does not exist"
Sep 17 18:30:05.000110 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:05.000036    2580 scope.go:117] "RemoveContainer" containerID="612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487"
Sep 17 18:30:05.000515 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:30:05.000375    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487\": container with ID starting with 612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487 not found: ID does not exist" containerID="612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487"
Sep 17 18:30:05.000515 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:05.000408    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487"} err="failed to get container status \"612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487\": rpc error: code = NotFound desc = could not find container \"612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487\": container with ID starting with 612faa83abe350937d45425614622e54766ae7d67e256445fa74998d267b1487 not found: ID does not exist"
Sep 17 18:30:05.001745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:05.001724    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"]
Sep 17 18:30:05.006099 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:05.006066    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8d64fdb-lq2zn"]
Sep 17 18:30:06.045428 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:06.045382    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" path="/var/lib/kubelet/pods/907bed2a-da20-4a04-bc5a-311d7a5f45a8/volumes"
Sep 17 18:30:08.614293 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:08.614260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.714862 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:08.714830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.815060 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:08.815034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.851797 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:14.851761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.916721 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:14.916691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.817165 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817133    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"]
Sep 17 18:30:16.817520 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817504    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main"
Sep 17 18:30:16.817520 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817518    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817525    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817531    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="storage-initializer"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817545    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="storage-initializer"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817557    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="storage-initializer"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817562    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="storage-initializer"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817572    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="llm-d-routing-sidecar"
Sep 17 18:30:16.817592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817577    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="llm-d-routing-sidecar"
Sep 17 18:30:16.817871 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817638    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="main"
Sep 17 18:30:16.817871 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817647    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="907bed2a-da20-4a04-bc5a-311d7a5f45a8" containerName="main"
Sep 17 18:30:16.817871 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.817654    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81402b7c-cd33-4055-b5ad-158e244ae347" containerName="llm-d-routing-sidecar"
Sep 17 18:30:16.821839 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.821817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.824562 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.824533    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 18:30:16.824695 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.824622    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 18:30:16.824829 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.824793    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 18:30:16.824829 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.824798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 18:30:16.824955 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.824873    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-8h7bc\""
Sep 17 18:30:16.824955 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.824942    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:30:16.832784 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.832761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"]
Sep 17 18:30:16.841610 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.841583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.958152 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.958317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4wz\" (UniqueName: \"kubernetes.io/projected/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kube-api-access-sz4wz\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.958317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.958317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-home\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.958317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-dshm\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.958606 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-model-cache\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.958606 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:16.958606 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:16.958455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.059644 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.059767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4wz\" (UniqueName: \"kubernetes.io/projected/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kube-api-access-sz4wz\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.059767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.059767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-home\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.059767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-dshm\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.059767 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-model-cache\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.060030 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.060030 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.059826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.060172 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.060147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.060219 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.060167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-home\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.060263 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.060216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-model-cache\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.060263 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.060242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.060551 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.060526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.062050 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.062028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-dshm\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.062370 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.062325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.070207 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.070160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4wz\" (UniqueName: \"kubernetes.io/projected/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kube-api-access-sz4wz\") pod \"lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.134147 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.134122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:17.414993 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:17.414969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"]
Sep 17 18:30:17.416944 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:30:17.416916    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e3f60a_460d_4ab2_a123_b1e5b6a4e0d1.slice/crio-0bf3cf9ea5c7e36bba7070f1d9de018d9e51c937499bc51600461a37e6e67134 WatchSource:0}: Error finding container 0bf3cf9ea5c7e36bba7070f1d9de018d9e51c937499bc51600461a37e6e67134: Status 404 returned error can't find the container with id 0bf3cf9ea5c7e36bba7070f1d9de018d9e51c937499bc51600461a37e6e67134
Sep 17 18:30:18.055624 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:18.055581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" event={"ID":"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1","Type":"ContainerStarted","Data":"5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b"}
Sep 17 18:30:18.055982 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:18.055639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" event={"ID":"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1","Type":"ContainerStarted","Data":"0bf3cf9ea5c7e36bba7070f1d9de018d9e51c937499bc51600461a37e6e67134"}
Sep 17 18:30:21.071757 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:21.071724    2580 generic.go:356] "Generic (PLEG): container finished" podID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerID="5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b" exitCode=0
Sep 17 18:30:21.072123 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:21.071795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" event={"ID":"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1","Type":"ContainerDied","Data":"5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b"}
Sep 17 18:30:22.080378 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:22.080344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" event={"ID":"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1","Type":"ContainerStarted","Data":"266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5"}
Sep 17 18:30:22.105210 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:22.105154    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podStartSLOduration=3.177458408 podStartE2EDuration="6.105137644s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.927679236s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:22.102527028 +0000 UTC m=+2998.441920619" watchObservedRunningTime="2026-09-17 18:30:22.105137644 +0000 UTC m=+2998.444531236"
Sep 17 18:30:22.135040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:22.135007    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:25.356365 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:25.356335    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:30:25.369842 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:25.369822    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:30:27.134683 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:27.134635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:30:27.136107 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:27.136060    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:30:29.888135 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:29.888076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:29.978988 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:29.978950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.142374 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:30.142341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.215176 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:30.215149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:33.756124 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:33.756071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.445207 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:36.445176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:37.135072 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:37.135027    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:30:42.343946 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:42.343908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:42.408097 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:42.408044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:47.135094 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:47.135037    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:30:57.135015 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:30:57.134973    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:31:07.134962 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:07.134910    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:31:17.135219 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:17.135123    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:31:20.673038 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.673004    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"]
Sep 17 18:31:20.676932 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.676908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.679724 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.679694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:31:20.679847 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.679721    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-qdtrb\""
Sep 17 18:31:20.679847 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.679757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 18:31:20.679847 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.679710    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:31:20.680050 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.680033    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:31:20.680218 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.680072    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:31:20.690387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.690362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"]
Sep 17 18:31:20.705754 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.705717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:20.769369 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4d22\" (UniqueName: \"kubernetes.io/projected/657db303-8341-4007-9bc6-27e6b3060331-kube-api-access-l4d22\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.769503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-home\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.769503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657db303-8341-4007-9bc6-27e6b3060331-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.769503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.769503 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.769641 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657db303-8341-4007-9bc6-27e6b3060331-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.769641 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.769641 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.769572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.870763 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.870908 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.870908 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4d22\" (UniqueName: \"kubernetes.io/projected/657db303-8341-4007-9bc6-27e6b3060331-kube-api-access-l4d22\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.870908 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-home\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.870908 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657db303-8341-4007-9bc6-27e6b3060331-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871112 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871112 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871112 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.870977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657db303-8341-4007-9bc6-27e6b3060331-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.871290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871438 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.871363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871598 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.871579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-home\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871689 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.871636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.871750 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.871709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657db303-8341-4007-9bc6-27e6b3060331-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.873296 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.873272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.873456 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.873435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657db303-8341-4007-9bc6-27e6b3060331-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.879428 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.879398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4d22\" (UniqueName: \"kubernetes.io/projected/657db303-8341-4007-9bc6-27e6b3060331-kube-api-access-l4d22\") pod \"router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:20.988522 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:20.988433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:21.273769 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:21.273736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"]
Sep 17 18:31:21.275953 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:31:21.275921    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657db303_8341_4007_9bc6_27e6b3060331.slice/crio-9c93123235af9620db61a83f3a3983aeb1950668eb1cb24dc93fbbcdb66de11e WatchSource:0}: Error finding container 9c93123235af9620db61a83f3a3983aeb1950668eb1cb24dc93fbbcdb66de11e: Status 404 returned error can't find the container with id 9c93123235af9620db61a83f3a3983aeb1950668eb1cb24dc93fbbcdb66de11e
Sep 17 18:31:21.295928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:21.295899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:21.379600 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:21.379571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:21.419522 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:21.419480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" event={"ID":"657db303-8341-4007-9bc6-27e6b3060331","Type":"ContainerStarted","Data":"21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751"}
Sep 17 18:31:21.419663 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:21.419592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" event={"ID":"657db303-8341-4007-9bc6-27e6b3060331","Type":"ContainerStarted","Data":"9c93123235af9620db61a83f3a3983aeb1950668eb1cb24dc93fbbcdb66de11e"}
Sep 17 18:31:24.442426 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:24.442394    2580 generic.go:356] "Generic (PLEG): container finished" podID="657db303-8341-4007-9bc6-27e6b3060331" containerID="21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751" exitCode=0
Sep 17 18:31:24.442845 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:24.442474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" event={"ID":"657db303-8341-4007-9bc6-27e6b3060331","Type":"ContainerDied","Data":"21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751"}
Sep 17 18:31:25.455138 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:25.455062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" event={"ID":"657db303-8341-4007-9bc6-27e6b3060331","Type":"ContainerStarted","Data":"ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba"}
Sep 17 18:31:25.482860 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:25.482779    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podStartSLOduration=3.211429951 podStartE2EDuration="5.482763313s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.271333362s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:25.481339652 +0000 UTC m=+3061.820733244" watchObservedRunningTime="2026-09-17 18:31:25.482763313 +0000 UTC m=+3061.822156902"
Sep 17 18:31:27.135275 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:27.135223    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:31:30.988876 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:30.988826    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:30.989380 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:30.988903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:31:30.990801 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:30.990766    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:31:37.144278 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:37.144244    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:31:37.152175 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:37.152140    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:31:40.989288 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:40.989237    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:31:50.989454 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:31:50.989406    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:32:00.989779 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:00.989734    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:32:04.575647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:04.575601    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58_13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1/storage-initializer/0.log"
Sep 17 18:32:04.586827 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:04.586791    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58_13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1/main/0.log"
Sep 17 18:32:05.639844 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:05.639809    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"]
Sep 17 18:32:05.655440 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:05.655410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:10.989525 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:10.989480    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:32:20.663270 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:20.663222    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" containerID="cri-o://266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5" gracePeriod=45
Sep 17 18:32:20.989861 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:20.989749    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:32:21.144974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.144939    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 18:32:21.328101 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.328061    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:32:21.499896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.499854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-cabundle-cert\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500128 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.499931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-model-cache\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500128 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.499958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-home\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500128 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.499991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tls-certs\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500128 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.500021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kserve-provision-location\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500128 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.500045    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-dshm\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500128 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.500077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tmp-dir\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500501 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.500137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz4wz\" (UniqueName: \"kubernetes.io/projected/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kube-api-access-sz4wz\") pod \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\" (UID: \"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1\") "
Sep 17 18:32:21.500569 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.500462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:21.500949 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.500905    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:21.500949 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.500928    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-home" (OuterVolumeSpecName: "home") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:21.502604 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.502573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-dshm" (OuterVolumeSpecName: "dshm") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:21.502856 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.502829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:21.502856 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.502836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kube-api-access-sz4wz" (OuterVolumeSpecName: "kube-api-access-sz4wz") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "kube-api-access-sz4wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:21.516159 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.516125    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:21.530928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.530901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" (UID: "13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:21.601308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601242    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.601308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601268    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.601308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601278    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.601308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601286    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz4wz\" (UniqueName: \"kubernetes.io/projected/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-kube-api-access-sz4wz\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.601308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601295    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.601308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601303    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.601308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601310    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.601617 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.601317    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:21.803560 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.803525    2580 generic.go:356] "Generic (PLEG): container finished" podID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerID="266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5" exitCode=0
Sep 17 18:32:21.804026 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.803583    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"
Sep 17 18:32:21.804026 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.803619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" event={"ID":"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1","Type":"ContainerDied","Data":"266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5"}
Sep 17 18:32:21.804026 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.803670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58" event={"ID":"13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1","Type":"ContainerDied","Data":"0bf3cf9ea5c7e36bba7070f1d9de018d9e51c937499bc51600461a37e6e67134"}
Sep 17 18:32:21.804026 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.803695    2580 scope.go:117] "RemoveContainer" containerID="266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5"
Sep 17 18:32:21.823582 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.823559    2580 scope.go:117] "RemoveContainer" containerID="5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b"
Sep 17 18:32:21.834936 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.834912    2580 scope.go:117] "RemoveContainer" containerID="266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5"
Sep 17 18:32:21.835497 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:32:21.835474    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5\": container with ID starting with 266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5 not found: ID does not exist" containerID="266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5"
Sep 17 18:32:21.835601 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.835507    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5"} err="failed to get container status \"266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5\": rpc error: code = NotFound desc = could not find container \"266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5\": container with ID starting with 266b749be621cd1f25fa21a4c430d388f5421250c5517267c723670a3ae985a5 not found: ID does not exist"
Sep 17 18:32:21.835601 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.835532    2580 scope.go:117] "RemoveContainer" containerID="5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b"
Sep 17 18:32:21.835828 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:32:21.835807    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b\": container with ID starting with 5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b not found: ID does not exist" containerID="5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b"
Sep 17 18:32:21.835918 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.835834    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b"} err="failed to get container status \"5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b\": rpc error: code = NotFound desc = could not find container \"5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b\": container with ID starting with 5fafc63cc40c9fd6ccbd0d21858d87a140f194051eb1e17a5b7df21d0b77bf0b not found: ID does not exist"
Sep 17 18:32:21.837775 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.837752    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"]
Sep 17 18:32:21.841139 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:32:21.841108    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e3f60a_460d_4ab2_a123_b1e5b6a4e0d1.slice/crio-0bf3cf9ea5c7e36bba7070f1d9de018d9e51c937499bc51600461a37e6e67134\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e3f60a_460d_4ab2_a123_b1e5b6a4e0d1.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 18:32:21.842349 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:21.842327    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6f9f7648bf-p7l58"]
Sep 17 18:32:22.046213 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:22.046172    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" path="/var/lib/kubelet/pods/13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1/volumes"
Sep 17 18:32:25.840340 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:25.840148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:25.940615 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:25.940581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:26.042297 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:26.042259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:26.158665 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:26.158635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:26.259900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:26.259864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:28.587457 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:28.587420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:28.689268 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:28.689235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:29.132600 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.132565    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"]
Sep 17 18:32:29.133041 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.133021    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="storage-initializer"
Sep 17 18:32:29.133041 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.133042    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="storage-initializer"
Sep 17 18:32:29.133171 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.133055    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main"
Sep 17 18:32:29.133171 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.133061    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main"
Sep 17 18:32:29.133171 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.133134    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e3f60a-460d-4ab2-a123-b1e5b6a4e0d1" containerName="main"
Sep 17 18:32:29.137725 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.137709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:29.141414 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.141382    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:32:29.141528 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.141435    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:32:29.141528 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.141480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-7h7dk\""
Sep 17 18:32:29.141652 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.141397    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:32:29.143421 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.143399    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"]
Sep 17 18:32:29.239630 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.239601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:29.263446 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.263419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct24g\" (UniqueName: \"kubernetes.io/projected/ca84c252-72d7-4423-a4bf-34bbf5e2fd65-kube-api-access-ct24g\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:29.364036 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.364007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct24g\" (UniqueName: \"kubernetes.io/projected/ca84c252-72d7-4423-a4bf-34bbf5e2fd65-kube-api-access-ct24g\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:29.372622 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:29.372598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct24g\" (UniqueName: \"kubernetes.io/projected/ca84c252-72d7-4423-a4bf-34bbf5e2fd65-kube-api-access-ct24g\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:30.989659 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:30.989612    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:32:32.626553 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:32.626517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:32.750437 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:32.750380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:35.361176 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:35.361134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:35.462613 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:35.462574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:37.126535 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.126453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:37.132137 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.132108    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:37.227471 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.227440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:37.227616 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.227560    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:37.230693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.230676    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") DevicePath \"csi-62f72874fdd373c6f2ecb37741d276550fed95538cb4f57f0372f6f151c9e395\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:37.653578 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.653541    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9768ff7dc0cbf26f9e7207856f3513c849778f5f6a05a7cdd4c15e40c8516187/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:37.667928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.667901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"e2e-pvc-lora-download-cbwzs\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:37.857035 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:37.856997    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:38.133541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:38.133510    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"]
Sep 17 18:32:38.134744 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:32:38.134710    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca84c252_72d7_4423_a4bf_34bbf5e2fd65.slice/crio-417725f17fd089c3cfbfac0cd0a1204b87de59adddf4aecb8d8717d2b52fe2cc WatchSource:0}: Error finding container 417725f17fd089c3cfbfac0cd0a1204b87de59adddf4aecb8d8717d2b52fe2cc: Status 404 returned error can't find the container with id 417725f17fd089c3cfbfac0cd0a1204b87de59adddf4aecb8d8717d2b52fe2cc
Sep 17 18:32:38.916338 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:38.916297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs" event={"ID":"ca84c252-72d7-4423-a4bf-34bbf5e2fd65","Type":"ContainerStarted","Data":"d9275d4a64bf17ec5c9b7854ff0b3b1c7e09a9faff53b95dc47250c0ab126de2"}
Sep 17 18:32:38.916538 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:38.916351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs" event={"ID":"ca84c252-72d7-4423-a4bf-34bbf5e2fd65","Type":"ContainerStarted","Data":"417725f17fd089c3cfbfac0cd0a1204b87de59adddf4aecb8d8717d2b52fe2cc"}
Sep 17 18:32:38.936577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:38.936511    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs" podStartSLOduration=12.936491406 podStartE2EDuration="12.936491406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:38.932715865 +0000 UTC m=+3135.272109459" watchObservedRunningTime="2026-09-17 18:32:38.936491406 +0000 UTC m=+3135.275885002"
Sep 17 18:32:40.930053 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:40.930020    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca84c252-72d7-4423-a4bf-34bbf5e2fd65" containerID="d9275d4a64bf17ec5c9b7854ff0b3b1c7e09a9faff53b95dc47250c0ab126de2" exitCode=0
Sep 17 18:32:40.930424 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:40.930109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs" event={"ID":"ca84c252-72d7-4423-a4bf-34bbf5e2fd65","Type":"ContainerDied","Data":"d9275d4a64bf17ec5c9b7854ff0b3b1c7e09a9faff53b95dc47250c0ab126de2"}
Sep 17 18:32:40.988786 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:40.988751    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:32:42.078247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.078221    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:42.269732 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.269648    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") "
Sep 17 18:32:42.269732 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.269688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct24g\" (UniqueName: \"kubernetes.io/projected/ca84c252-72d7-4423-a4bf-34bbf5e2fd65-kube-api-access-ct24g\") pod \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\" (UID: \"ca84c252-72d7-4423-a4bf-34bbf5e2fd65\") "
Sep 17 18:32:42.272038 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.272007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca84c252-72d7-4423-a4bf-34bbf5e2fd65-kube-api-access-ct24g" (OuterVolumeSpecName: "kube-api-access-ct24g") pod "ca84c252-72d7-4423-a4bf-34bbf5e2fd65" (UID: "ca84c252-72d7-4423-a4bf-34bbf5e2fd65"). InnerVolumeSpecName "kube-api-access-ct24g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:42.273436 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.273414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5" (OuterVolumeSpecName: "model-storage") pod "ca84c252-72d7-4423-a4bf-34bbf5e2fd65" (UID: "ca84c252-72d7-4423-a4bf-34bbf5e2fd65"). InnerVolumeSpecName "pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:32:42.371228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.371189    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct24g\" (UniqueName: \"kubernetes.io/projected/ca84c252-72d7-4423-a4bf-34bbf5e2fd65-kube-api-access-ct24g\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:32:42.371387 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.371246    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") on node \"ip-10-0-139-39.ec2.internal\" "
Sep 17 18:32:42.942981 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.942954    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"
Sep 17 18:32:42.943178 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.942991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs" event={"ID":"ca84c252-72d7-4423-a4bf-34bbf5e2fd65","Type":"ContainerDied","Data":"417725f17fd089c3cfbfac0cd0a1204b87de59adddf4aecb8d8717d2b52fe2cc"}
Sep 17 18:32:42.943178 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:42.943027    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="417725f17fd089c3cfbfac0cd0a1204b87de59adddf4aecb8d8717d2b52fe2cc"
Sep 17 18:32:43.342197 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:43.342172    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5") on node "ip-10-0-139-39.ec2.internal"
Sep 17 18:32:43.379115 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:43.379061    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"csi-62f72874fdd373c6f2ecb37741d276550fed95538cb4f57f0372f6f151c9e395\""
Sep 17 18:32:50.989318 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:50.989276    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:32:53.796693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.796657    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"]
Sep 17 18:32:53.797213 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.797193    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca84c252-72d7-4423-a4bf-34bbf5e2fd65" containerName="storage-initializer"
Sep 17 18:32:53.797303 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.797216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca84c252-72d7-4423-a4bf-34bbf5e2fd65" containerName="storage-initializer"
Sep 17 18:32:53.797366 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.797332    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca84c252-72d7-4423-a4bf-34bbf5e2fd65" containerName="storage-initializer"
Sep 17 18:32:53.801737 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.801702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.804630 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.804579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:32:53.804768 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.804669    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:32:53.804768 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.804686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:32:53.804889 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.804786    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:32:53.804889 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.804690    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:32:53.805144 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.805119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-7h7dk\""
Sep 17 18:32:53.816279 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.816255    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"]
Sep 17 18:32:53.839565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.839535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:53.862533 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-dshm\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.862644 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-home\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.862644 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.862725 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.862725 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.862725 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8277df4d-28d4-4c6d-a290-4ca1975b37f2-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.862725 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndncr\" (UniqueName: \"kubernetes.io/projected/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kube-api-access-ndncr\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.862878 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.862773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964053 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-dshm\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-home\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8277df4d-28d4-4c6d-a290-4ca1975b37f2-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndncr\" (UniqueName: \"kubernetes.io/projected/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kube-api-access-ndncr\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964528 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964584 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-home\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964639 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.964972 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.965067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.964986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.965566 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.965544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8277df4d-28d4-4c6d-a290-4ca1975b37f2-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.966408 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.966386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-dshm\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.966689 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.966671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:53.972840 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:53.972820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndncr\" (UniqueName: \"kubernetes.io/projected/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kube-api-access-ndncr\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.387724 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.387675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.393417 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.393387    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.488954 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.488919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.489139 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.489062    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.492054 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.492029    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") DevicePath \"csi-62f72874fdd373c6f2ecb37741d276550fed95538cb4f57f0372f6f151c9e395\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.584604 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.584562    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9768ff7dc0cbf26f9e7207856f3513c849778f5f6a05a7cdd4c15e40c8516187/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.604208 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.604181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.716610 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.716570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:32:57.760905 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:57.760822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:58.000803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:58.000770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"]
Sep 17 18:32:58.001387 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:32:58.001358    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8277df4d_28d4_4c6d_a290_4ca1975b37f2.slice/crio-f12430e6cc808c92e3dec48d0c4dfa9779b8781a7ee9bf88ebf338a443f1e02b WatchSource:0}: Error finding container f12430e6cc808c92e3dec48d0c4dfa9779b8781a7ee9bf88ebf338a443f1e02b: Status 404 returned error can't find the container with id f12430e6cc808c92e3dec48d0c4dfa9779b8781a7ee9bf88ebf338a443f1e02b
Sep 17 18:32:58.026259 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:58.026231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" event={"ID":"8277df4d-28d4-4c6d-a290-4ca1975b37f2","Type":"ContainerStarted","Data":"f12430e6cc808c92e3dec48d0c4dfa9779b8781a7ee9bf88ebf338a443f1e02b"}
Sep 17 18:32:59.035359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:32:59.035312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" event={"ID":"8277df4d-28d4-4c6d-a290-4ca1975b37f2","Type":"ContainerStarted","Data":"26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af"}
Sep 17 18:33:00.989711 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:00.989656    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:33:01.050045 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:01.050014    2580 generic.go:356] "Generic (PLEG): container finished" podID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerID="26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af" exitCode=0
Sep 17 18:33:01.050213 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:01.050097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" event={"ID":"8277df4d-28d4-4c6d-a290-4ca1975b37f2","Type":"ContainerDied","Data":"26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af"}
Sep 17 18:33:02.058310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:02.058272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" event={"ID":"8277df4d-28d4-4c6d-a290-4ca1975b37f2","Type":"ContainerStarted","Data":"b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1"}
Sep 17 18:33:02.083623 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:02.083568    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podStartSLOduration=6.698779093 podStartE2EDuration="9.083551988s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.384772895s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:02.080348565 +0000 UTC m=+3158.419742150" watchObservedRunningTime="2026-09-17 18:33:02.083551988 +0000 UTC m=+3158.422945579"
Sep 17 18:33:02.717287 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:02.717244    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:33:07.717572 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:07.717536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:33:07.719509 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:07.719472    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 18:33:10.989242 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:10.989185    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 18:33:17.717289 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:17.717242    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 18:33:20.999597 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:20.999564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:33:21.007389 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:21.007367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:33:27.717368 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:27.717325    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 18:33:29.768295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:29.768262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:29.802666 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:29.802624    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"]
Sep 17 18:33:29.867855 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:29.867817    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:37.717607 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:37.717563    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 18:33:44.826058 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:44.826008    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main" containerID="cri-o://ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba" gracePeriod=45
Sep 17 18:33:45.588659 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.588629    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:33:45.720333 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-home\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.720541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4d22\" (UniqueName: \"kubernetes.io/projected/657db303-8341-4007-9bc6-27e6b3060331-kube-api-access-l4d22\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.720541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-dshm\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.720541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-tmp-dir\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.720541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657db303-8341-4007-9bc6-27e6b3060331-tls-certs\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.720541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657db303-8341-4007-9bc6-27e6b3060331-cabundle-cert\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.720541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720526    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-kserve-provision-location\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.720963 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657db303-8341-4007-9bc6-27e6b3060331-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:45.721078 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.721043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-model-cache" (OuterVolumeSpecName: "model-cache") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.721222 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.720555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-model-cache\") pod \"657db303-8341-4007-9bc6-27e6b3060331\" (UID: \"657db303-8341-4007-9bc6-27e6b3060331\") "
Sep 17 18:33:45.721312 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.721199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-home" (OuterVolumeSpecName: "home") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.721628 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.721568    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657db303-8341-4007-9bc6-27e6b3060331-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.721628 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.721589    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.721628 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.721603    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.723353 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.723326    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657db303-8341-4007-9bc6-27e6b3060331-kube-api-access-l4d22" (OuterVolumeSpecName: "kube-api-access-l4d22") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "kube-api-access-l4d22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:45.723470 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.723383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-dshm" (OuterVolumeSpecName: "dshm") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.723560 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.723539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657db303-8341-4007-9bc6-27e6b3060331-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:45.743488 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.743437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.746208 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.746174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "657db303-8341-4007-9bc6-27e6b3060331" (UID: "657db303-8341-4007-9bc6-27e6b3060331"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:45.823010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.822974    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.823010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.823007    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.823010 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.823016    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657db303-8341-4007-9bc6-27e6b3060331-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.823267 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.823026    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657db303-8341-4007-9bc6-27e6b3060331-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:45.823267 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:45.823034    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4d22\" (UniqueName: \"kubernetes.io/projected/657db303-8341-4007-9bc6-27e6b3060331-kube-api-access-l4d22\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:33:46.322985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.322880    2580 generic.go:356] "Generic (PLEG): container finished" podID="657db303-8341-4007-9bc6-27e6b3060331" containerID="ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba" exitCode=0
Sep 17 18:33:46.322985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.322942    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"
Sep 17 18:33:46.322985 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.322950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" event={"ID":"657db303-8341-4007-9bc6-27e6b3060331","Type":"ContainerDied","Data":"ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba"}
Sep 17 18:33:46.323602 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.323012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j" event={"ID":"657db303-8341-4007-9bc6-27e6b3060331","Type":"ContainerDied","Data":"9c93123235af9620db61a83f3a3983aeb1950668eb1cb24dc93fbbcdb66de11e"}
Sep 17 18:33:46.323602 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.323037    2580 scope.go:117] "RemoveContainer" containerID="ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba"
Sep 17 18:33:46.334714 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.334690    2580 scope.go:117] "RemoveContainer" containerID="21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751"
Sep 17 18:33:46.342052 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.342021    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"]
Sep 17 18:33:46.345440 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.345411    2580 scope.go:117] "RemoveContainer" containerID="ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba"
Sep 17 18:33:46.345876 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:33:46.345841    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba\": container with ID starting with ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba not found: ID does not exist" containerID="ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba"
Sep 17 18:33:46.345960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.345882    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba"} err="failed to get container status \"ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba\": rpc error: code = NotFound desc = could not find container \"ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba\": container with ID starting with ae06ef3840132088d31539baf6b565d418b7c1db2e845d005ba59f2e736e25ba not found: ID does not exist"
Sep 17 18:33:46.345960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.345910    2580 scope.go:117] "RemoveContainer" containerID="21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751"
Sep 17 18:33:46.346599 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:33:46.346571    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751\": container with ID starting with 21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751 not found: ID does not exist" containerID="21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751"
Sep 17 18:33:46.346674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.346606    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751"} err="failed to get container status \"21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751\": rpc error: code = NotFound desc = could not find container \"21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751\": container with ID starting with 21fed1e37cc19356e32464b1a52dace099b21f3c5d995ef18e78925b6be00751 not found: ID does not exist"
Sep 17 18:33:46.348072 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:46.348052    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-564b69b5ff-tk48j"]
Sep 17 18:33:47.717346 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:47.717293    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 18:33:48.048109 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:48.048010    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="657db303-8341-4007-9bc6-27e6b3060331" path="/var/lib/kubelet/pods/657db303-8341-4007-9bc6-27e6b3060331/volumes"
Sep 17 18:33:57.717668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:33:57.717627    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 18:34:07.717715 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:07.717669    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 18:34:17.728195 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:17.728079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:34:17.736281 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:17.736252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:34:19.949979 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:19.949933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:20.041573 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:20.041533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:25.049212 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:25.049173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:28.781136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:28.781057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:43.898235 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:43.898200    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv_8277df4d-28d4-4c6d-a290-4ca1975b37f2/storage-initializer/0.log"
Sep 17 18:34:43.908699 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:43.908663    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv_8277df4d-28d4-4c6d-a290-4ca1975b37f2/main/0.log"
Sep 17 18:34:44.881485 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:44.881442    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"]
Sep 17 18:34:44.898239 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:44.898204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:46.882452 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:46.882413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:47.797714 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:47.797663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:53.595382 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:53.595341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:59.904778 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:34:59.904736    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main" containerID="cri-o://b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1" gracePeriod=45
Sep 17 18:35:00.557854 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.557830    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:35:00.668632 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.668748 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-model-cache\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.668748 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-home\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.668748 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tls-certs\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.668872 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-dshm\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.668872 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kserve-provision-location\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.668872 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tmp-dir\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.669028 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndncr\" (UniqueName: \"kubernetes.io/projected/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kube-api-access-ndncr\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.669028 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668950    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8277df4d-28d4-4c6d-a290-4ca1975b37f2-cabundle-cert\") pod \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\" (UID: \"8277df4d-28d4-4c6d-a290-4ca1975b37f2\") "
Sep 17 18:35:00.669028 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.668997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:00.669242 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.669152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-home" (OuterVolumeSpecName: "home") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:00.669388 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.669365    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.669459 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.669397    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.669459 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.669371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8277df4d-28d4-4c6d-a290-4ca1975b37f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:00.671385 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.671153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kube-api-access-ndncr" (OuterVolumeSpecName: "kube-api-access-ndncr") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "kube-api-access-ndncr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:00.671385 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.671310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:00.671551 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.671395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-dshm" (OuterVolumeSpecName: "dshm") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:00.673546 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.673520    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:35:00.683290 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.683265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:00.694489 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.694464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8277df4d-28d4-4c6d-a290-4ca1975b37f2" (UID: "8277df4d-28d4-4c6d-a290-4ca1975b37f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:00.762000 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.761975    2580 generic.go:356] "Generic (PLEG): container finished" podID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerID="b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1" exitCode=0
Sep 17 18:35:00.762108 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.762026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"
Sep 17 18:35:00.762108 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.762059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" event={"ID":"8277df4d-28d4-4c6d-a290-4ca1975b37f2","Type":"ContainerDied","Data":"b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1"}
Sep 17 18:35:00.762189 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.762136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv" event={"ID":"8277df4d-28d4-4c6d-a290-4ca1975b37f2","Type":"ContainerDied","Data":"f12430e6cc808c92e3dec48d0c4dfa9779b8781a7ee9bf88ebf338a443f1e02b"}
Sep 17 18:35:00.762189 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.762162    2580 scope.go:117] "RemoveContainer" containerID="b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1"
Sep 17 18:35:00.770109 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.770058    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndncr\" (UniqueName: \"kubernetes.io/projected/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kube-api-access-ndncr\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.770109 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.770106    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8277df4d-28d4-4c6d-a290-4ca1975b37f2-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.770234 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.770139    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") on node \"ip-10-0-139-39.ec2.internal\" "
Sep 17 18:35:00.770234 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.770156    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.770234 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.770170    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.770234 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.770179    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.770234 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.770187    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8277df4d-28d4-4c6d-a290-4ca1975b37f2-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:00.771696 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.771677    2580 scope.go:117] "RemoveContainer" containerID="26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af"
Sep 17 18:35:00.780165 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.780143    2580 scope.go:117] "RemoveContainer" containerID="b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1"
Sep 17 18:35:00.780308 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.780289    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5") on node "ip-10-0-139-39.ec2.internal"
Sep 17 18:35:00.780468 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:35:00.780450    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1\": container with ID starting with b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1 not found: ID does not exist" containerID="b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1"
Sep 17 18:35:00.780514 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.780474    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1"} err="failed to get container status \"b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1\": rpc error: code = NotFound desc = could not find container \"b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1\": container with ID starting with b052159c6ecf9c3d76dfd152432a2765d954f5fef6e81ce0a1362e047e661ed1 not found: ID does not exist"
Sep 17 18:35:00.780514 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.780490    2580 scope.go:117] "RemoveContainer" containerID="26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af"
Sep 17 18:35:00.780759 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:35:00.780742    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af\": container with ID starting with 26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af not found: ID does not exist" containerID="26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af"
Sep 17 18:35:00.780812 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.780762    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af"} err="failed to get container status \"26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af\": rpc error: code = NotFound desc = could not find container \"26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af\": container with ID starting with 26fa02b62359319360c9781afa31722614a5a22d4945d14da8cee6e2b4f369af not found: ID does not exist"
Sep 17 18:35:00.786257 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.786234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"]
Sep 17 18:35:00.789363 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.789341    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f677774f6-mv2mv"]
Sep 17 18:35:00.870905 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:00.870884    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-eb5a5b58-c4a4-44a0-aaa2-5d96a191b436\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09cec4dd99397c6a5\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"csi-62f72874fdd373c6f2ecb37741d276550fed95538cb4f57f0372f6f151c9e395\""
Sep 17 18:35:02.046126 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:02.046061    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" path="/var/lib/kubelet/pods/8277df4d-28d4-4c6d-a290-4ca1975b37f2/volumes"
Sep 17 18:35:05.018403 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:05.018370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.122316 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:05.122285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.219425 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:05.219389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.297259 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:05.297187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.353978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:05.353944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.579928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:07.579882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.633075 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:07.633044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:08.313266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313208    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"]
Sep 17 18:35:08.313650 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313636    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="storage-initializer"
Sep 17 18:35:08.313697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313651    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="storage-initializer"
Sep 17 18:35:08.313697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313665    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="storage-initializer"
Sep 17 18:35:08.313697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313672    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="storage-initializer"
Sep 17 18:35:08.313697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313680    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main"
Sep 17 18:35:08.313697 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313687    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main"
Sep 17 18:35:08.313858 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313701    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main"
Sep 17 18:35:08.313858 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313707    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main"
Sep 17 18:35:08.313858 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313768    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8277df4d-28d4-4c6d-a290-4ca1975b37f2" containerName="main"
Sep 17 18:35:08.313858 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.313778    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="657db303-8341-4007-9bc6-27e6b3060331" containerName="main"
Sep 17 18:35:08.325954 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.325920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:08.326758 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.326714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"]
Sep 17 18:35:08.328924 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.328894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 18:35:08.329040 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.328991    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:35:08.329126 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.329049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:35:08.330262 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.330233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-6924w\""
Sep 17 18:35:08.384596 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.384569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:08.425818 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.425792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsrb\" (UniqueName: \"kubernetes.io/projected/8c41c5e3-ba9f-4f2e-91e7-acc18944c574-kube-api-access-wmsrb\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:08.526889 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.526863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsrb\" (UniqueName: \"kubernetes.io/projected/8c41c5e3-ba9f-4f2e-91e7-acc18944c574-kube-api-access-wmsrb\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:08.535260 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:08.535239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsrb\" (UniqueName: \"kubernetes.io/projected/8c41c5e3-ba9f-4f2e-91e7-acc18944c574-kube-api-access-wmsrb\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:09.998551 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:09.998519    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"]
Sep 17 18:35:10.002034 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.002009    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cbwzs"]
Sep 17 18:35:10.045505 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.045478    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca84c252-72d7-4423-a4bf-34bbf5e2fd65" path="/var/lib/kubelet/pods/ca84c252-72d7-4423-a4bf-34bbf5e2fd65/volumes"
Sep 17 18:35:10.564259 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.564226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.664526 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.664493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.743318 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.743293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:10.748541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.748517    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:10.844664 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.844608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:10.844755 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.844736    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:10.847435 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:10.847414    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") DevicePath \"csi-ab99a9d84dd79f985d33929d7f73116958793a11c0ebfee908e9d6b8a8323d7c\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:11.044443 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.044410    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b29ad4c9233b0319cb9fc48887268e13ec34bffa72f5b1e6312944b269e58a31/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:11.058642 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.058616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"e2e-pvc-lora-download-fszlp\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:11.278827 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.278795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:11.349227 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.349203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:11.620161 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.620131    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"]
Sep 17 18:35:11.621674 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:35:11.621651    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c41c5e3_ba9f_4f2e_91e7_acc18944c574.slice/crio-971ed952b882a3cc737f38ffa1cd77f155b5a58fbb1d10cdeab323b59efe5c0a WatchSource:0}: Error finding container 971ed952b882a3cc737f38ffa1cd77f155b5a58fbb1d10cdeab323b59efe5c0a: Status 404 returned error can't find the container with id 971ed952b882a3cc737f38ffa1cd77f155b5a58fbb1d10cdeab323b59efe5c0a
Sep 17 18:35:11.829116 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.829061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp" event={"ID":"8c41c5e3-ba9f-4f2e-91e7-acc18944c574","Type":"ContainerStarted","Data":"219b44738b27795ee7f4eb197a76433f7836a2bf5874d61aa09475c1aa856584"}
Sep 17 18:35:11.829116 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.829116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp" event={"ID":"8c41c5e3-ba9f-4f2e-91e7-acc18944c574","Type":"ContainerStarted","Data":"971ed952b882a3cc737f38ffa1cd77f155b5a58fbb1d10cdeab323b59efe5c0a"}
Sep 17 18:35:11.845212 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:11.845171    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp" podStartSLOduration=6.845157037 podStartE2EDuration="6.845157037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:11.843877081 +0000 UTC m=+3288.183270670" watchObservedRunningTime="2026-09-17 18:35:11.845157037 +0000 UTC m=+3288.184550629"
Sep 17 18:35:13.845008 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:13.844920    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c41c5e3-ba9f-4f2e-91e7-acc18944c574" containerID="219b44738b27795ee7f4eb197a76433f7836a2bf5874d61aa09475c1aa856584" exitCode=0
Sep 17 18:35:13.845459 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:13.845002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp" event={"ID":"8c41c5e3-ba9f-4f2e-91e7-acc18944c574","Type":"ContainerDied","Data":"219b44738b27795ee7f4eb197a76433f7836a2bf5874d61aa09475c1aa856584"}
Sep 17 18:35:15.009013 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.008986    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:15.077206 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.077181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmsrb\" (UniqueName: \"kubernetes.io/projected/8c41c5e3-ba9f-4f2e-91e7-acc18944c574-kube-api-access-wmsrb\") pod \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") "
Sep 17 18:35:15.077331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.077300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\" (UID: \"8c41c5e3-ba9f-4f2e-91e7-acc18944c574\") "
Sep 17 18:35:15.079383 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.079359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c41c5e3-ba9f-4f2e-91e7-acc18944c574-kube-api-access-wmsrb" (OuterVolumeSpecName: "kube-api-access-wmsrb") pod "8c41c5e3-ba9f-4f2e-91e7-acc18944c574" (UID: "8c41c5e3-ba9f-4f2e-91e7-acc18944c574"). InnerVolumeSpecName "kube-api-access-wmsrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:15.079803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.079781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f" (OuterVolumeSpecName: "model-storage") pod "8c41c5e3-ba9f-4f2e-91e7-acc18944c574" (UID: "8c41c5e3-ba9f-4f2e-91e7-acc18944c574"). InnerVolumeSpecName "pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:35:15.178781 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.178755    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsrb\" (UniqueName: \"kubernetes.io/projected/8c41c5e3-ba9f-4f2e-91e7-acc18944c574-kube-api-access-wmsrb\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:35:15.178896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.178799    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") on node \"ip-10-0-139-39.ec2.internal\" "
Sep 17 18:35:15.859025 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.858991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"
Sep 17 18:35:15.859211 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.859015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp" event={"ID":"8c41c5e3-ba9f-4f2e-91e7-acc18944c574","Type":"ContainerDied","Data":"971ed952b882a3cc737f38ffa1cd77f155b5a58fbb1d10cdeab323b59efe5c0a"}
Sep 17 18:35:15.859211 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:15.859062    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="971ed952b882a3cc737f38ffa1cd77f155b5a58fbb1d10cdeab323b59efe5c0a"
Sep 17 18:35:16.148203 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:16.148183    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f") on node "ip-10-0-139-39.ec2.internal"
Sep 17 18:35:16.187297 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:16.187262    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"csi-ab99a9d84dd79f985d33929d7f73116958793a11c0ebfee908e9d6b8a8323d7c\""
Sep 17 18:35:17.341354 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:17.341309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.440247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:17.440216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.725628 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:17.725593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.790515 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:17.790463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:18.069544 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:18.069507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:25.404194 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:25.404157    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:35:25.416344 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:25.416320    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:35:27.593248 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:27.593213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:27.655341 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:27.655314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:50.587804 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.587730    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"]
Sep 17 18:35:50.588253 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.588169    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c41c5e3-ba9f-4f2e-91e7-acc18944c574" containerName="storage-initializer"
Sep 17 18:35:50.588253 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.588183    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c41c5e3-ba9f-4f2e-91e7-acc18944c574" containerName="storage-initializer"
Sep 17 18:35:50.588253 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.588252    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c41c5e3-ba9f-4f2e-91e7-acc18944c574" containerName="storage-initializer"
Sep 17 18:35:50.591316 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.591297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.593864 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.593838    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:35:50.594016 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.593995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-6924w\""
Sep 17 18:35:50.594067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.594029    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 18:35:50.594142 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.594072    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:35:50.603528 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.603507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"]
Sep 17 18:35:50.735693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.735661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.735693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.735694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.735869 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.735763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.735869 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.735785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmjzc\" (UniqueName: \"kubernetes.io/projected/d3079b60-affe-45fc-bffa-86c1ad8646c0-kube-api-access-kmjzc\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.735869 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.735805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.735869 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.735822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3079b60-affe-45fc-bffa-86c1ad8646c0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837009 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.836980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmjzc\" (UniqueName: \"kubernetes.io/projected/d3079b60-affe-45fc-bffa-86c1ad8646c0-kube-api-access-kmjzc\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3079b60-affe-45fc-bffa-86c1ad8646c0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837385 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837525 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837588 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.837588 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.837580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.839979 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.839925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.840163 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.840142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3079b60-affe-45fc-bffa-86c1ad8646c0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:50.845190 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:50.845168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmjzc\" (UniqueName: \"kubernetes.io/projected/d3079b60-affe-45fc-bffa-86c1ad8646c0-kube-api-access-kmjzc\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:35:51.368295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:35:51.368265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:00.609594 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:00.609555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:00.614972 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:00.614944    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:00.710518 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:00.710493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:00.710658 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:00.710636    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:00.713635 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:00.713614    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") DevicePath \"csi-ab99a9d84dd79f985d33929d7f73116958793a11c0ebfee908e9d6b8a8323d7c\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:00.801417 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:00.801387    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b29ad4c9233b0319cb9fc48887268e13ec34bffa72f5b1e6312944b269e58a31/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:00.821177 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:00.821156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-csldz\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:01.102599 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:01.102560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:01.381577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:01.381551    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"]
Sep 17 18:36:01.383394 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:36:01.383366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3079b60_affe_45fc_bffa_86c1ad8646c0.slice/crio-82a768fe2e0fd657cefde616ac29fad459382dd1b80420b9f899f7f2dbfced90 WatchSource:0}: Error finding container 82a768fe2e0fd657cefde616ac29fad459382dd1b80420b9f899f7f2dbfced90: Status 404 returned error can't find the container with id 82a768fe2e0fd657cefde616ac29fad459382dd1b80420b9f899f7f2dbfced90
Sep 17 18:36:02.118100 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:02.118052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" event={"ID":"d3079b60-affe-45fc-bffa-86c1ad8646c0","Type":"ContainerStarted","Data":"0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35"}
Sep 17 18:36:02.118451 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:02.118128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" event={"ID":"d3079b60-affe-45fc-bffa-86c1ad8646c0","Type":"ContainerStarted","Data":"82a768fe2e0fd657cefde616ac29fad459382dd1b80420b9f899f7f2dbfced90"}
Sep 17 18:36:02.140530 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:02.140463    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podStartSLOduration=12.140443596 podStartE2EDuration="12.140443596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:02.137471787 +0000 UTC m=+3338.476865379" watchObservedRunningTime="2026-09-17 18:36:02.140443596 +0000 UTC m=+3338.479837190"
Sep 17 18:36:03.102973 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:03.102938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:11.102963 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:11.102927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:36:11.104310 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:11.104282    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:36:21.103723 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:21.103687    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:36:31.103264 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:31.103218    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:36:35.167028 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:35.166983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:41.103487 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:41.103433    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:36:45.129451 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:45.129398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:51.103553 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:36:51.103507    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:37:01.103104 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:01.103050    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:37:05.183919 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:05.183875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.281750 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:05.281704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:11.103671 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:11.103627    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:37:11.401551 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:11.401516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:17.135454 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:17.135349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:17.273577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:17.273537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:21.112469 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:21.112433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:37:21.119570 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:21.119547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:37:22.300616 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.300578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:22.488500 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.488463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:22.613372 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.613282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:22.648910 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.648880    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:37:22.653138 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.653118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.655797 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.655770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:37:22.655797 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.655784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 18:37:22.656775 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.656755    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-4994h\""
Sep 17 18:37:22.656775 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.656769    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 18:37:22.656949 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.656789    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 18:37:22.656949 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.656795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 18:37:22.656949 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.656824    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-ndff7\""
Sep 17 18:37:22.663238 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.663210    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:37:22.692760 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.692714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:22.768164 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbb75bab-b674-4504-a896-53318c58b3d5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.768272 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb75bab-b674-4504-a896-53318c58b3d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.768272 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.768378 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.768378 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.768456 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xg2l\" (UniqueName: \"kubernetes.io/projected/bbb75bab-b674-4504-a896-53318c58b3d5-kube-api-access-4xg2l\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.768456 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.768554 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.768466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869066 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xg2l\" (UniqueName: \"kubernetes.io/projected/bbb75bab-b674-4504-a896-53318c58b3d5-kube-api-access-4xg2l\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbb75bab-b674-4504-a896-53318c58b3d5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb75bab-b674-4504-a896-53318c58b3d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869581 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869689 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869689 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869783 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.869848 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.869826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbb75bab-b674-4504-a896-53318c58b3d5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.871330 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.871306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.871549 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.871529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb75bab-b674-4504-a896-53318c58b3d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.876515 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.876498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xg2l\" (UniqueName: \"kubernetes.io/projected/bbb75bab-b674-4504-a896-53318c58b3d5-kube-api-access-4xg2l\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:22.964715 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:22.964691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:23.241187 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:23.241160    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:37:23.578993 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:23.578911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bbb75bab-b674-4504-a896-53318c58b3d5","Type":"ContainerStarted","Data":"2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe"}
Sep 17 18:37:23.578993 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:23.578970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bbb75bab-b674-4504-a896-53318c58b3d5","Type":"ContainerStarted","Data":"16f83c57e9f55b087e4144d3fc50f9d826878eb76fa1d4982f080fea8734e77c"}
Sep 17 18:37:25.591410 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:25.591380    2580 generic.go:356] "Generic (PLEG): container finished" podID="bbb75bab-b674-4504-a896-53318c58b3d5" containerID="2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe" exitCode=0
Sep 17 18:37:25.591712 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:25.591452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bbb75bab-b674-4504-a896-53318c58b3d5","Type":"ContainerDied","Data":"2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe"}
Sep 17 18:37:26.599813 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:26.599778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bbb75bab-b674-4504-a896-53318c58b3d5","Type":"ContainerStarted","Data":"e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68"}
Sep 17 18:37:26.628435 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:26.628375    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.459055005 podStartE2EDuration="4.628353234s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.169298229s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:26.6235929 +0000 UTC m=+3422.962986506" watchObservedRunningTime="2026-09-17 18:37:26.628353234 +0000 UTC m=+3422.967746824"
Sep 17 18:37:26.965429 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:26.965393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:32.965140 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:32.965078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:37:32.966695 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:32.966665    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:37:36.105311 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:36.105273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:42.965739 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:42.965693    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:37:52.965611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:37:52.965560    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:38:02.792046 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:02.792013    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-csldz_d3079b60-affe-45fc-bffa-86c1ad8646c0/main/0.log"
Sep 17 18:38:02.965521 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:02.965484    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:38:03.722457 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:03.722416    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"]
Sep 17 18:38:03.753456 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:03.753423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:12.965403 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:12.965353    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:38:18.746574 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:18.746522    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" containerID="cri-o://0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35" gracePeriod=45
Sep 17 18:38:19.113264 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.113177    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 18:38:19.615571 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.615547    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:38:19.755685 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.755617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-home\") pod \"d3079b60-affe-45fc-bffa-86c1ad8646c0\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") "
Sep 17 18:38:19.755685 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.755663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-tmp-dir\") pod \"d3079b60-affe-45fc-bffa-86c1ad8646c0\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") "
Sep 17 18:38:19.756169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.755707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmjzc\" (UniqueName: \"kubernetes.io/projected/d3079b60-affe-45fc-bffa-86c1ad8646c0-kube-api-access-kmjzc\") pod \"d3079b60-affe-45fc-bffa-86c1ad8646c0\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") "
Sep 17 18:38:19.756169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.755736    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-model-cache\") pod \"d3079b60-affe-45fc-bffa-86c1ad8646c0\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") "
Sep 17 18:38:19.756169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.755868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") pod \"d3079b60-affe-45fc-bffa-86c1ad8646c0\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") "
Sep 17 18:38:19.756169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.755927    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-dshm\") pod \"d3079b60-affe-45fc-bffa-86c1ad8646c0\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") "
Sep 17 18:38:19.756169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.755951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3079b60-affe-45fc-bffa-86c1ad8646c0-tls-certs\") pod \"d3079b60-affe-45fc-bffa-86c1ad8646c0\" (UID: \"d3079b60-affe-45fc-bffa-86c1ad8646c0\") "
Sep 17 18:38:19.756169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.756048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3079b60-affe-45fc-bffa-86c1ad8646c0" (UID: "d3079b60-affe-45fc-bffa-86c1ad8646c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:19.756169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.756077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-home" (OuterVolumeSpecName: "home") pod "d3079b60-affe-45fc-bffa-86c1ad8646c0" (UID: "d3079b60-affe-45fc-bffa-86c1ad8646c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:19.756501 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.756234    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:38:19.756501 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.756251    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:38:19.758595 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.758563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-dshm" (OuterVolumeSpecName: "dshm") pod "d3079b60-affe-45fc-bffa-86c1ad8646c0" (UID: "d3079b60-affe-45fc-bffa-86c1ad8646c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:19.758699 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.758566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3079b60-affe-45fc-bffa-86c1ad8646c0-kube-api-access-kmjzc" (OuterVolumeSpecName: "kube-api-access-kmjzc") pod "d3079b60-affe-45fc-bffa-86c1ad8646c0" (UID: "d3079b60-affe-45fc-bffa-86c1ad8646c0"). InnerVolumeSpecName "kube-api-access-kmjzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:19.758812 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.758779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3079b60-affe-45fc-bffa-86c1ad8646c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3079b60-affe-45fc-bffa-86c1ad8646c0" (UID: "d3079b60-affe-45fc-bffa-86c1ad8646c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:19.763749 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.763727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f" (OuterVolumeSpecName: "kserve-pvc-source") pod "d3079b60-affe-45fc-bffa-86c1ad8646c0" (UID: "d3079b60-affe-45fc-bffa-86c1ad8646c0"). InnerVolumeSpecName "pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:38:19.767839 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.767804    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3079b60-affe-45fc-bffa-86c1ad8646c0" (UID: "d3079b60-affe-45fc-bffa-86c1ad8646c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:19.857364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.857333    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") on node \"ip-10-0-139-39.ec2.internal\" "
Sep 17 18:38:19.857364 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.857364    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:38:19.857497 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.857375    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3079b60-affe-45fc-bffa-86c1ad8646c0-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:38:19.857497 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.857384    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3079b60-affe-45fc-bffa-86c1ad8646c0-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:38:19.857497 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.857398    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmjzc\" (UniqueName: \"kubernetes.io/projected/d3079b60-affe-45fc-bffa-86c1ad8646c0-kube-api-access-kmjzc\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:38:19.888180 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.888164    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f") on node "ip-10-0-139-39.ec2.internal"
Sep 17 18:38:19.913782 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.913756    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerID="0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35" exitCode=0
Sep 17 18:38:19.913874 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.913854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"
Sep 17 18:38:19.913915 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.913868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" event={"ID":"d3079b60-affe-45fc-bffa-86c1ad8646c0","Type":"ContainerDied","Data":"0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35"}
Sep 17 18:38:19.913950 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.913923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz" event={"ID":"d3079b60-affe-45fc-bffa-86c1ad8646c0","Type":"ContainerDied","Data":"82a768fe2e0fd657cefde616ac29fad459382dd1b80420b9f899f7f2dbfced90"}
Sep 17 18:38:19.913983 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.913957    2580 scope.go:117] "RemoveContainer" containerID="0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35"
Sep 17 18:38:19.923729 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.923712    2580 scope.go:117] "RemoveContainer" containerID="0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35"
Sep 17 18:38:19.923987 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:38:19.923969    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35\": container with ID starting with 0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35 not found: ID does not exist" containerID="0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35"
Sep 17 18:38:19.924051 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.923995    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35"} err="failed to get container status \"0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35\": rpc error: code = NotFound desc = could not find container \"0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35\": container with ID starting with 0b7363272101ff204051cd18b6850875221dfb68123f0bf8401466d2bc12ac35 not found: ID does not exist"
Sep 17 18:38:19.938414 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.938383    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"]
Sep 17 18:38:19.945933 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.943736    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-csldz"]
Sep 17 18:38:19.958678 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:19.958648    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-525dcf84-e120-4b2c-a49d-7d3306aeea93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01764e96685c6767f\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"csi-ab99a9d84dd79f985d33929d7f73116958793a11c0ebfee908e9d6b8a8323d7c\""
Sep 17 18:38:20.046227 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:20.046174    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" path="/var/lib/kubelet/pods/d3079b60-affe-45fc-bffa-86c1ad8646c0/volumes"
Sep 17 18:38:22.965907 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:22.965861    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:38:23.796242 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:23.796203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:23.897049 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:23.897003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:24.000415 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:24.000348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:28.274634 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.274553    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"]
Sep 17 18:38:28.275159 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.275139    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main"
Sep 17 18:38:28.275247 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.275162    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main"
Sep 17 18:38:28.275301 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.275283    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3079b60-affe-45fc-bffa-86c1ad8646c0" containerName="main"
Sep 17 18:38:28.279623 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.279601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.282428 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.282333    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 18:38:28.282428 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.282375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 18:38:28.282428 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.282416    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-c7h8l\""
Sep 17 18:38:28.282428 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.282426    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:38:28.282672 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.282471    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 18:38:28.282730 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.282713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 18:38:28.290008 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.289978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"]
Sep 17 18:38:28.297575 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.297547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:28.425978 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.425946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hs4q\" (UniqueName: \"kubernetes.io/projected/fe52fb44-14fc-4238-867e-634777fc62c1-kube-api-access-6hs4q\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.426121 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.425983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-home\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.426121 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.426003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-tmp-dir\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.426121 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.426021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-dshm\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.426238 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.426129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe52fb44-14fc-4238-867e-634777fc62c1-tls-certs\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.426238 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.426175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-kserve-provision-location\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.426238 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.426194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-model-cache\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.426331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.426264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe52fb44-14fc-4238-867e-634777fc62c1-cabundle-cert\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.501899 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.501858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:28.526808 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.526726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-model-cache\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.526937 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.526803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe52fb44-14fc-4238-867e-634777fc62c1-cabundle-cert\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.526937 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.526878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hs4q\" (UniqueName: \"kubernetes.io/projected/fe52fb44-14fc-4238-867e-634777fc62c1-kube-api-access-6hs4q\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.526937 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.526913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-home\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.526941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-tmp-dir\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.526972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-dshm\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.527013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe52fb44-14fc-4238-867e-634777fc62c1-tls-certs\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.527058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-kserve-provision-location\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527389 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.527167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-model-cache\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527389 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.527340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-home\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527523 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.527422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-kserve-provision-location\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527581 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.527547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-tmp-dir\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.527939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.527914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe52fb44-14fc-4238-867e-634777fc62c1-cabundle-cert\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.529320 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.529293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-dshm\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.529698 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.529679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe52fb44-14fc-4238-867e-634777fc62c1-tls-certs\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.534962 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.534939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hs4q\" (UniqueName: \"kubernetes.io/projected/fe52fb44-14fc-4238-867e-634777fc62c1-kube-api-access-6hs4q\") pod \"tls-verification-test-kserve-7bc985b5f6-lm8tr\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.590541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.590514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:28.786304 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.786226    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"]
Sep 17 18:38:28.790366 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.790338    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fszlp"]
Sep 17 18:38:28.871254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.871219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"]
Sep 17 18:38:28.872935 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:38:28.872908    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe52fb44_14fc_4238_867e_634777fc62c1.slice/crio-9d4266a3256911bb9369780deeca437381a9bb90ba9a6bfc695ba9fc006a72a0 WatchSource:0}: Error finding container 9d4266a3256911bb9369780deeca437381a9bb90ba9a6bfc695ba9fc006a72a0: Status 404 returned error can't find the container with id 9d4266a3256911bb9369780deeca437381a9bb90ba9a6bfc695ba9fc006a72a0
Sep 17 18:38:28.967389 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.967361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" event={"ID":"fe52fb44-14fc-4238-867e-634777fc62c1","Type":"ContainerStarted","Data":"b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d"}
Sep 17 18:38:28.967558 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:28.967415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" event={"ID":"fe52fb44-14fc-4238-867e-634777fc62c1","Type":"ContainerStarted","Data":"9d4266a3256911bb9369780deeca437381a9bb90ba9a6bfc695ba9fc006a72a0"}
Sep 17 18:38:30.049216 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:30.049181    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c41c5e3-ba9f-4f2e-91e7-acc18944c574" path="/var/lib/kubelet/pods/8c41c5e3-ba9f-4f2e-91e7-acc18944c574/volumes"
Sep 17 18:38:30.098910 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:30.098872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:30.199179 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:30.199143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:30.299776 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:30.299682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:30.400693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:30.400652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:31.089898 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:31.089865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:31.988866 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:31.988831    2580 generic.go:356] "Generic (PLEG): container finished" podID="fe52fb44-14fc-4238-867e-634777fc62c1" containerID="b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d" exitCode=0
Sep 17 18:38:31.989034 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:31.988912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" event={"ID":"fe52fb44-14fc-4238-867e-634777fc62c1","Type":"ContainerDied","Data":"b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d"}
Sep 17 18:38:32.965488 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:32.965444    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:38:32.997799 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:32.997769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" event={"ID":"fe52fb44-14fc-4238-867e-634777fc62c1","Type":"ContainerStarted","Data":"c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237"}
Sep 17 18:38:33.021804 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:33.021748    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podStartSLOduration=2.76120168 podStartE2EDuration="5.021730092s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.260528412s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:33.01963483 +0000 UTC m=+3489.359028421" watchObservedRunningTime="2026-09-17 18:38:33.021730092 +0000 UTC m=+3489.361123683"
Sep 17 18:38:33.591155 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:33.591123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:36.798167 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:36.798120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:36.867807 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:36.867773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:38.590869 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:38.590834    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:38:38.592783 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:38.592750    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:38:42.965634 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:42.965584    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:38:44.491647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:44.491549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:48.591449 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:48.591408    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:38:52.965385 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:52.965346    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:38:58.591794 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:38:58.591748    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:39:02.974752 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:02.974716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:39:02.982052 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:02.982027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:39:07.753971 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:07.753935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:07.784326 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:07.784293    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:39:07.818510 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:07.818468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:08.590997 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:08.590956    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:39:10.362216 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:10.362184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:18.591519 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:18.591479    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:39:20.950619 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:20.950583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:22.813889 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:22.813823    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" containerID="cri-o://e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68" gracePeriod=45
Sep 17 18:39:22.976258 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:22.976217    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 18:39:23.565611 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.565586    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:39:23.698434 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbb75bab-b674-4504-a896-53318c58b3d5-cabundle-cert\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698434 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-model-cache\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-tmp-dir\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-home\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-kserve-provision-location\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb75bab-b674-4504-a896-53318c58b3d5-tls-certs\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xg2l\" (UniqueName: \"kubernetes.io/projected/bbb75bab-b674-4504-a896-53318c58b3d5-kube-api-access-4xg2l\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698982 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-dshm\") pod \"bbb75bab-b674-4504-a896-53318c58b3d5\" (UID: \"bbb75bab-b674-4504-a896-53318c58b3d5\") "
Sep 17 18:39:23.698982 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:23.698982 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb75bab-b674-4504-a896-53318c58b3d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:23.698982 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698942    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbb75bab-b674-4504-a896-53318c58b3d5-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.698982 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.698958    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.702053 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.701783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb75bab-b674-4504-a896-53318c58b3d5-kube-api-access-4xg2l" (OuterVolumeSpecName: "kube-api-access-4xg2l") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "kube-api-access-4xg2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:23.703969 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.703943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-home" (OuterVolumeSpecName: "home") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:23.719844 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.719815    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:23.720484 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.720451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-dshm" (OuterVolumeSpecName: "dshm") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:23.720592 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.720571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb75bab-b674-4504-a896-53318c58b3d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:23.731433 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.731407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbb75bab-b674-4504-a896-53318c58b3d5" (UID: "bbb75bab-b674-4504-a896-53318c58b3d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:23.799582 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.799561    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.799668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.799585    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.799668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.799596    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb75bab-b674-4504-a896-53318c58b3d5-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.799668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.799607    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xg2l\" (UniqueName: \"kubernetes.io/projected/bbb75bab-b674-4504-a896-53318c58b3d5-kube-api-access-4xg2l\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.799668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.799614    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.799668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:23.799627    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb75bab-b674-4504-a896-53318c58b3d5-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:39:24.317948 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.317865    2580 generic.go:356] "Generic (PLEG): container finished" podID="bbb75bab-b674-4504-a896-53318c58b3d5" containerID="e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68" exitCode=0
Sep 17 18:39:24.317948 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.317923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:39:24.318450 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.317954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bbb75bab-b674-4504-a896-53318c58b3d5","Type":"ContainerDied","Data":"e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68"}
Sep 17 18:39:24.318450 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.318049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"bbb75bab-b674-4504-a896-53318c58b3d5","Type":"ContainerDied","Data":"16f83c57e9f55b087e4144d3fc50f9d826878eb76fa1d4982f080fea8734e77c"}
Sep 17 18:39:24.318450 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.318107    2580 scope.go:117] "RemoveContainer" containerID="e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68"
Sep 17 18:39:24.328401 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.328383    2580 scope.go:117] "RemoveContainer" containerID="2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe"
Sep 17 18:39:24.337425 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.337403    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:39:24.337898 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.337881    2580 scope.go:117] "RemoveContainer" containerID="e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68"
Sep 17 18:39:24.338183 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:39:24.338158    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68\": container with ID starting with e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68 not found: ID does not exist" containerID="e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68"
Sep 17 18:39:24.338289 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.338188    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68"} err="failed to get container status \"e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68\": rpc error: code = NotFound desc = could not find container \"e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68\": container with ID starting with e0ba5ea9a51661d167b1c79bf5c91a58a3c91ded4abff8c65490fd3c24e92c68 not found: ID does not exist"
Sep 17 18:39:24.338289 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.338206    2580 scope.go:117] "RemoveContainer" containerID="2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe"
Sep 17 18:39:24.338717 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:39:24.338658    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe\": container with ID starting with 2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe not found: ID does not exist" containerID="2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe"
Sep 17 18:39:24.338717 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.338691    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe"} err="failed to get container status \"2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe\": rpc error: code = NotFound desc = could not find container \"2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe\": container with ID starting with 2fa6909e2cb69c3a0873c39b1bcf1c9394b110c4fda560e32226d7b49eea7fbe not found: ID does not exist"
Sep 17 18:39:24.340961 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:24.340935    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:39:25.402373 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:25.402341    2580 scope.go:117] "RemoveContainer" containerID="d9275d4a64bf17ec5c9b7854ff0b3b1c7e09a9faff53b95dc47250c0ab126de2"
Sep 17 18:39:26.045499 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:26.045463    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" path="/var/lib/kubelet/pods/bbb75bab-b674-4504-a896-53318c58b3d5/volumes"
Sep 17 18:39:28.591701 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:28.591654    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:39:38.590909 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:38.590870    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:39:48.590849 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:48.590799    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:39:57.913602 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:57.913557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:58.591533 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:58.591488    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:39:59.671414 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:39:59.671380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:04.271166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:04.271126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:05.731647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:05.731615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:05.820154 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:05.820119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:08.591526 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:08.591485    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 18:40:10.145443 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:10.145409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:10.208880 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:10.208844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:16.237200 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:16.237106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:18.601047 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:18.601017    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:40:18.608348 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:18.608325    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:40:19.023845 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:19.023805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:23.595454 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:23.595408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:23.611526 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:23.611493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"]
Sep 17 18:40:23.653155 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:23.653111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:25.441805 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:25.441778    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:40:25.455191 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:25.455168    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:40:34.602438 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:34.602403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:38.636168 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:38.636114    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main" containerID="cri-o://c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237" gracePeriod=45
Sep 17 18:40:39.283202 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.283179    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:40:39.404683 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.404659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-home\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.404800 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.404699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-dshm\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.404800 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.404724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe52fb44-14fc-4238-867e-634777fc62c1-cabundle-cert\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.404800 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.404761    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-tmp-dir\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.404800 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.404793    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hs4q\" (UniqueName: \"kubernetes.io/projected/fe52fb44-14fc-4238-867e-634777fc62c1-kube-api-access-6hs4q\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.405007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.404809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe52fb44-14fc-4238-867e-634777fc62c1-tls-certs\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.405007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.404915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-kserve-provision-location\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.405007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.405002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-model-cache\") pod \"fe52fb44-14fc-4238-867e-634777fc62c1\" (UID: \"fe52fb44-14fc-4238-867e-634777fc62c1\") "
Sep 17 18:40:39.405195 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.405135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe52fb44-14fc-4238-867e-634777fc62c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:39.405380 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.405254    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe52fb44-14fc-4238-867e-634777fc62c1-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.405380 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.405306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-home" (OuterVolumeSpecName: "home") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:39.405380 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.405341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:39.407166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.407134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe52fb44-14fc-4238-867e-634777fc62c1-kube-api-access-6hs4q" (OuterVolumeSpecName: "kube-api-access-6hs4q") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "kube-api-access-6hs4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:39.407391 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.407369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe52fb44-14fc-4238-867e-634777fc62c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:39.407473 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.407406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-dshm" (OuterVolumeSpecName: "dshm") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:39.426474 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.426445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:39.437021 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.437000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe52fb44-14fc-4238-867e-634777fc62c1" (UID: "fe52fb44-14fc-4238-867e-634777fc62c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:39.506037 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.506015    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.506037 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.506039    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.506037 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.506047    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.506235 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.506055    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.506235 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.506065    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hs4q\" (UniqueName: \"kubernetes.io/projected/fe52fb44-14fc-4238-867e-634777fc62c1-kube-api-access-6hs4q\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.506235 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.506075    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe52fb44-14fc-4238-867e-634777fc62c1-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.506235 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.506104    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe52fb44-14fc-4238-867e-634777fc62c1-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:40:39.638988 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.638939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:39.699731 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.699600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:39.747496 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.747446    2580 generic.go:356] "Generic (PLEG): container finished" podID="fe52fb44-14fc-4238-867e-634777fc62c1" containerID="c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237" exitCode=0
Sep 17 18:40:39.747670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.747522    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"
Sep 17 18:40:39.747670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.747525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" event={"ID":"fe52fb44-14fc-4238-867e-634777fc62c1","Type":"ContainerDied","Data":"c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237"}
Sep 17 18:40:39.747670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.747649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr" event={"ID":"fe52fb44-14fc-4238-867e-634777fc62c1","Type":"ContainerDied","Data":"9d4266a3256911bb9369780deeca437381a9bb90ba9a6bfc695ba9fc006a72a0"}
Sep 17 18:40:39.747852 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.747682    2580 scope.go:117] "RemoveContainer" containerID="c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237"
Sep 17 18:40:39.758169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.758145    2580 scope.go:117] "RemoveContainer" containerID="b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d"
Sep 17 18:40:39.767882 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.767867    2580 scope.go:117] "RemoveContainer" containerID="c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237"
Sep 17 18:40:39.768271 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:40:39.768245    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237\": container with ID starting with c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237 not found: ID does not exist" containerID="c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237"
Sep 17 18:40:39.768376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.768281    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237"} err="failed to get container status \"c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237\": rpc error: code = NotFound desc = could not find container \"c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237\": container with ID starting with c7695fbcf937421aa901cf3e03771c2deba26235dcd5ad542b98ab9027966237 not found: ID does not exist"
Sep 17 18:40:39.768376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.768303    2580 scope.go:117] "RemoveContainer" containerID="b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d"
Sep 17 18:40:39.768614 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:40:39.768593    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d\": container with ID starting with b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d not found: ID does not exist" containerID="b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d"
Sep 17 18:40:39.768702 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.768618    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d"} err="failed to get container status \"b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d\": rpc error: code = NotFound desc = could not find container \"b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d\": container with ID starting with b82c039e26ce674b979058ac41cfc6007f32e14e6c849e16a21f46308b9b318d not found: ID does not exist"
Sep 17 18:40:39.775056 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.775013    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"]
Sep 17 18:40:39.780862 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:39.780839    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bc985b5f6-lm8tr"]
Sep 17 18:40:40.046239 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:40:40.046161    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" path="/var/lib/kubelet/pods/fe52fb44-14fc-4238-867e-634777fc62c1/volumes"
Sep 17 18:41:13.709577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:13.709536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:13.803532 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:13.803494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:16.903145 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.903109    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"]
Sep 17 18:41:16.904407 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.904376    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="storage-initializer"
Sep 17 18:41:16.904565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.904553    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="storage-initializer"
Sep 17 18:41:16.904693 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.904682    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="storage-initializer"
Sep 17 18:41:16.904765 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.904757    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="storage-initializer"
Sep 17 18:41:16.904844 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.904835    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main"
Sep 17 18:41:16.904912 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.904903    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main"
Sep 17 18:41:16.904984 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.904976    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main"
Sep 17 18:41:16.905053 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.905046    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main"
Sep 17 18:41:16.905312 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.905293    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb75bab-b674-4504-a896-53318c58b3d5" containerName="main"
Sep 17 18:41:16.905454 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.905441    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe52fb44-14fc-4238-867e-634777fc62c1" containerName="main"
Sep 17 18:41:16.910118 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.910097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.914052 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.914029    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:41:16.914236 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.914213    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:41:16.914663 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.914287    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 18:41:16.914663 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.914459    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:41:16.914663 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.914503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4t7nz\""
Sep 17 18:41:16.915147 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.915129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:41:16.919261 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.919239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"]
Sep 17 18:41:16.971328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-kserve-provision-location\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.971468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-dshm\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.971468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bbc904-4cca-442e-b2ea-337847a0a119-cabundle-cert\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.971468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-home\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.971604 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-model-cache\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.971604 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbc904-4cca-442e-b2ea-337847a0a119-tls-certs\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.971604 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5zz\" (UniqueName: \"kubernetes.io/projected/04bbc904-4cca-442e-b2ea-337847a0a119-kube-api-access-cw5zz\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.971604 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.971591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-tmp-dir\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:16.972061 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:16.972043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:17.072131 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bbc904-4cca-442e-b2ea-337847a0a119-cabundle-cert\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-home\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-model-cache\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbc904-4cca-442e-b2ea-337847a0a119-tls-certs\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072379 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5zz\" (UniqueName: \"kubernetes.io/projected/04bbc904-4cca-442e-b2ea-337847a0a119-kube-api-access-cw5zz\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072458 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-tmp-dir\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072514 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-kserve-provision-location\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072566 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-home\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072620 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-dshm\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072720 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-model-cache\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072831 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-tmp-dir\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072901 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-kserve-provision-location\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.072901 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.072877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bbc904-4cca-442e-b2ea-337847a0a119-cabundle-cert\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.074560 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.074541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-dshm\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.074884 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.074867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbc904-4cca-442e-b2ea-337847a0a119-tls-certs\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.083048 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.083024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5zz\" (UniqueName: \"kubernetes.io/projected/04bbc904-4cca-442e-b2ea-337847a0a119-kube-api-access-cw5zz\") pod \"prestop-hook-test-kserve-78dbc984b4-vqtfd\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.150431 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.150400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:17.220741 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.220671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:17.500161 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.500135    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"]
Sep 17 18:41:17.500851 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:41:17.500824    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bbc904_4cca_442e_b2ea_337847a0a119.slice/crio-9b9d22a22ac879359156829bc1925d95f68752cdf12bc3f8f337dd5df53bc7fe WatchSource:0}: Error finding container 9b9d22a22ac879359156829bc1925d95f68752cdf12bc3f8f337dd5df53bc7fe: Status 404 returned error can't find the container with id 9b9d22a22ac879359156829bc1925d95f68752cdf12bc3f8f337dd5df53bc7fe
Sep 17 18:41:17.958080 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.958041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" event={"ID":"04bbc904-4cca-442e-b2ea-337847a0a119","Type":"ContainerStarted","Data":"f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84"}
Sep 17 18:41:17.958612 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:17.958113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" event={"ID":"04bbc904-4cca-442e-b2ea-337847a0a119","Type":"ContainerStarted","Data":"9b9d22a22ac879359156829bc1925d95f68752cdf12bc3f8f337dd5df53bc7fe"}
Sep 17 18:41:19.970580 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:19.970546    2580 generic.go:356] "Generic (PLEG): container finished" podID="04bbc904-4cca-442e-b2ea-337847a0a119" containerID="f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84" exitCode=0
Sep 17 18:41:19.970928 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:19.970618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" event={"ID":"04bbc904-4cca-442e-b2ea-337847a0a119","Type":"ContainerDied","Data":"f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84"}
Sep 17 18:41:20.499343 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:20.499296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:20.978092 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:20.978043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" event={"ID":"04bbc904-4cca-442e-b2ea-337847a0a119","Type":"ContainerStarted","Data":"0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66"}
Sep 17 18:41:21.003338 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:21.003293    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podStartSLOduration=2.813942349 podStartE2EDuration="5.003279603s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.189337254s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:21.000985847 +0000 UTC m=+3657.340379477" watchObservedRunningTime="2026-09-17 18:41:21.003279603 +0000 UTC m=+3657.342673192"
Sep 17 18:41:21.221404 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:21.221370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:25.436348 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:25.436315    2580 scope.go:117] "RemoveContainer" containerID="219b44738b27795ee7f4eb197a76433f7836a2bf5874d61aa09475c1aa856584"
Sep 17 18:41:26.353212 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:26.353171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:26.484989 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:26.484948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:26.563423 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:26.563388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:27.221403 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:27.221361    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:41:27.222932 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:27.222903    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:41:29.681726 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:29.681679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:29.779430 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:29.779394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:33.582682 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.582643    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"]
Sep 17 18:41:33.585683 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.585661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.589318 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.589296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 18:41:33.589536 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.589311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dgjhj\""
Sep 17 18:41:33.589776 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.589321    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 18:41:33.589918 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.589553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 18:41:33.595623 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.595598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"]
Sep 17 18:41:33.622299 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.622254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:33.707942 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.707907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46z97\" (UniqueName: \"kubernetes.io/projected/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-kube-api-access-46z97\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.708106 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.707994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.708106 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.708056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.708198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.708112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.708198 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.708184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.708272 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.708230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.809641 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.809607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.809641 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.809651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.809877 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.809672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.809877 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.809725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.809877 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.809769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.809877 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.809808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46z97\" (UniqueName: \"kubernetes.io/projected/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-kube-api-access-46z97\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.810152 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.810127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.810228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.810161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.810228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.810174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.812204 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.812177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.812314 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.812295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.822146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.822122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46z97\" (UniqueName: \"kubernetes.io/projected/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-kube-api-access-46z97\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:33.875161 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.875041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:33.898939 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:33.898915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:34.171636 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:34.171600    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"]
Sep 17 18:41:34.173641 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:41:34.173609    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4899d7bc_df43_4d90_bfd9_ff70fba02ba4.slice/crio-1162938ccd106596a71749786e7b4f17d358b48ae829a539ff1095842d3095de WatchSource:0}: Error finding container 1162938ccd106596a71749786e7b4f17d358b48ae829a539ff1095842d3095de: Status 404 returned error can't find the container with id 1162938ccd106596a71749786e7b4f17d358b48ae829a539ff1095842d3095de
Sep 17 18:41:35.063908 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:35.063870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" event={"ID":"4899d7bc-df43-4d90-bfd9-ff70fba02ba4","Type":"ContainerStarted","Data":"ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887"}
Sep 17 18:41:35.064315 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:35.063933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" event={"ID":"4899d7bc-df43-4d90-bfd9-ff70fba02ba4","Type":"ContainerStarted","Data":"1162938ccd106596a71749786e7b4f17d358b48ae829a539ff1095842d3095de"}
Sep 17 18:41:35.899803 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:35.899768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:36.525540 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:36.525503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:37.221400 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:37.221362    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:41:42.360302 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:42.360266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:42.424677 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:42.424638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:43.899148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:43.899109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:43.911714 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:43.911690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:43.931107 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:43.931036    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" podStartSLOduration=10.931021758 podStartE2EDuration="10.931021758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:35.080621715 +0000 UTC m=+3671.420015307" watchObservedRunningTime="2026-09-17 18:41:43.931021758 +0000 UTC m=+3680.270415347"
Sep 17 18:41:44.122986 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:44.122952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:41:47.221888 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:47.221851    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:41:48.297180 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:48.297136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:57.221865 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:41:57.221824    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:42:07.221036 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.220990    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:42:07.431329 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.431278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"]
Sep 17 18:42:07.434845 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.434817    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:07.443694 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:42:07.443659    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:42:07.443694 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:07.443694 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:07.443694 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887" cmd=["/bin/sleep","15"]
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.443718    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" message="" err=<
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:42:07.443734    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" podUID="4899d7bc-df43-4d90-bfd9-ff70fba02ba4" containerName="main" containerID="cri-o://ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887"
Sep 17 18:42:07.444012 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.443792    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" podUID="4899d7bc-df43-4d90-bfd9-ff70fba02ba4" containerName="main" containerID="cri-o://ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887" gracePeriod=60
Sep 17 18:42:07.533419 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.533317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:07.709393 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.709368    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:42:07.800848 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.800765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tls-certs\") pod \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") "
Sep 17 18:42:07.801007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.800862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-home\") pod \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") "
Sep 17 18:42:07.801007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.800917    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-model-cache\") pod \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") "
Sep 17 18:42:07.801007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.800944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46z97\" (UniqueName: \"kubernetes.io/projected/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-kube-api-access-46z97\") pod \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") "
Sep 17 18:42:07.801007 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.800968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tmp-dir\") pod \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") "
Sep 17 18:42:07.801254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.801007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-dshm\") pod \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\" (UID: \"4899d7bc-df43-4d90-bfd9-ff70fba02ba4\") "
Sep 17 18:42:07.801254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.801193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4899d7bc-df43-4d90-bfd9-ff70fba02ba4" (UID: "4899d7bc-df43-4d90-bfd9-ff70fba02ba4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:07.801254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.801215    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-home" (OuterVolumeSpecName: "home") pod "4899d7bc-df43-4d90-bfd9-ff70fba02ba4" (UID: "4899d7bc-df43-4d90-bfd9-ff70fba02ba4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:07.801407 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.801304    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:42:07.801407 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.801300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4899d7bc-df43-4d90-bfd9-ff70fba02ba4" (UID: "4899d7bc-df43-4d90-bfd9-ff70fba02ba4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:07.803359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.803328    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4899d7bc-df43-4d90-bfd9-ff70fba02ba4" (UID: "4899d7bc-df43-4d90-bfd9-ff70fba02ba4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:07.803359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.803331    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-dshm" (OuterVolumeSpecName: "dshm") pod "4899d7bc-df43-4d90-bfd9-ff70fba02ba4" (UID: "4899d7bc-df43-4d90-bfd9-ff70fba02ba4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:07.803359 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.803333    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-kube-api-access-46z97" (OuterVolumeSpecName: "kube-api-access-46z97") pod "4899d7bc-df43-4d90-bfd9-ff70fba02ba4" (UID: "4899d7bc-df43-4d90-bfd9-ff70fba02ba4"). InnerVolumeSpecName "kube-api-access-46z97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:07.902035 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.901996    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:42:07.902035 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.902028    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:42:07.902035 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.902037    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:42:07.902292 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.902047    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46z97\" (UniqueName: \"kubernetes.io/projected/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-kube-api-access-46z97\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:42:07.902292 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:07.902057    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4899d7bc-df43-4d90-bfd9-ff70fba02ba4-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.263746 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.263711    2580 generic.go:356] "Generic (PLEG): container finished" podID="4899d7bc-df43-4d90-bfd9-ff70fba02ba4" containerID="ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887" exitCode=0
Sep 17 18:42:08.264209 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.263757    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"
Sep 17 18:42:08.264209 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.263797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" event={"ID":"4899d7bc-df43-4d90-bfd9-ff70fba02ba4","Type":"ContainerDied","Data":"ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887"}
Sep 17 18:42:08.264209 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.263867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk" event={"ID":"4899d7bc-df43-4d90-bfd9-ff70fba02ba4","Type":"ContainerDied","Data":"1162938ccd106596a71749786e7b4f17d358b48ae829a539ff1095842d3095de"}
Sep 17 18:42:08.264209 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.263903    2580 scope.go:117] "RemoveContainer" containerID="ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887"
Sep 17 18:42:08.273810 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.273795    2580 scope.go:117] "RemoveContainer" containerID="ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887"
Sep 17 18:42:08.274131 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:42:08.274108    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887\": container with ID starting with ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887 not found: ID does not exist" containerID="ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887"
Sep 17 18:42:08.274244 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.274136    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887"} err="failed to get container status \"ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887\": rpc error: code = NotFound desc = could not find container \"ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887\": container with ID starting with ee4ea8830aaecb2563f5405a5cc8a45c5b5a893f5d619eeb96116076306ba887 not found: ID does not exist"
Sep 17 18:42:08.280324 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.280300    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"]
Sep 17 18:42:08.283474 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:08.283453    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fltvdk"]
Sep 17 18:42:10.047079 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:10.047042    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4899d7bc-df43-4d90-bfd9-ff70fba02ba4" path="/var/lib/kubelet/pods/4899d7bc-df43-4d90-bfd9-ff70fba02ba4/volumes"
Sep 17 18:42:17.221820 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:17.221770    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:42:27.221574 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:27.221527    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:42:37.221464 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:37.221415    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:42:47.221745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:47.221703    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:42:57.221276 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:57.221234    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 17 18:42:57.480147 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:57.480050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:57.573817 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:42:57.573770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:03.842317 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:03.842275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:06.589652 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.589619    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"]
Sep 17 18:43:06.590225 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.590204    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4899d7bc-df43-4d90-bfd9-ff70fba02ba4" containerName="main"
Sep 17 18:43:06.590322 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.590226    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4899d7bc-df43-4d90-bfd9-ff70fba02ba4" containerName="main"
Sep 17 18:43:06.590379 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.590337    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4899d7bc-df43-4d90-bfd9-ff70fba02ba4" containerName="main"
Sep 17 18:43:06.595039 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.595016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.598969 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.598948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 18:43:06.599106 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.599026    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-dpmjb\""
Sep 17 18:43:06.599106 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.599096    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 18:43:06.599227 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.599185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 18:43:06.603599 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.603577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"]
Sep 17 18:43:06.617009 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.616978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:06.693849 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.693815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.694032 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.693861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw828\" (UniqueName: \"kubernetes.io/projected/242a7208-73fa-4d72-a34d-422d344609f4-kube-api-access-nw828\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.694032 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.693890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.694032 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.693944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242a7208-73fa-4d72-a34d-422d344609f4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.694032 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.693967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.694250 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.694026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795209 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242a7208-73fa-4d72-a34d-422d344609f4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795411 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw828\" (UniqueName: \"kubernetes.io/projected/242a7208-73fa-4d72-a34d-422d344609f4-kube-api-access-nw828\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795705 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795764 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.795938 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.795916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.797867 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.797846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.798583 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.798558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242a7208-73fa-4d72-a34d-422d344609f4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.804311 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.804291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw828\" (UniqueName: \"kubernetes.io/projected/242a7208-73fa-4d72-a34d-422d344609f4-kube-api-access-nw828\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:06.898223 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.898188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:06.908118 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:06.908074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:07.187074 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:07.187039    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"]
Sep 17 18:43:07.188319 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:43:07.188284    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242a7208_73fa_4d72_a34d_422d344609f4.slice/crio-dfc6eeaf8dc4c440829a8549ebd102813e75b1fea3eea5a4822c3b58441180b0 WatchSource:0}: Error finding container dfc6eeaf8dc4c440829a8549ebd102813e75b1fea3eea5a4822c3b58441180b0: Status 404 returned error can't find the container with id dfc6eeaf8dc4c440829a8549ebd102813e75b1fea3eea5a4822c3b58441180b0
Sep 17 18:43:07.233370 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:07.233345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:43:07.241968 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:07.241945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:43:07.597577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:07.597476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" event={"ID":"242a7208-73fa-4d72-a34d-422d344609f4","Type":"ContainerStarted","Data":"050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0"}
Sep 17 18:43:07.597577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:07.597546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" event={"ID":"242a7208-73fa-4d72-a34d-422d344609f4","Type":"ContainerStarted","Data":"dfc6eeaf8dc4c440829a8549ebd102813e75b1fea3eea5a4822c3b58441180b0"}
Sep 17 18:43:07.615222 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:07.614678    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" podStartSLOduration=1.614661158 podStartE2EDuration="1.614661158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:07.614019616 +0000 UTC m=+3763.953413209" watchObservedRunningTime="2026-09-17 18:43:07.614661158 +0000 UTC m=+3763.954054751"
Sep 17 18:43:07.908896 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:07.908861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:09.512671 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:09.512628    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"]
Sep 17 18:43:10.211973 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:10.211937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:10.280215 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:10.280181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:12.618388 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:12.618353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:16.909075 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:16.908994    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:16.921597 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:16.921569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:17.664295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:17.664265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:24.535545 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:24.535504    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" containerID="cri-o://0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66" gracePeriod=45
Sep 17 18:43:25.286702 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.286670    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:43:25.348621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-model-cache\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.348621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw5zz\" (UniqueName: \"kubernetes.io/projected/04bbc904-4cca-442e-b2ea-337847a0a119-kube-api-access-cw5zz\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.348840 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348723    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-dshm\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.348840 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bbc904-4cca-442e-b2ea-337847a0a119-cabundle-cert\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.348840 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-kserve-provision-location\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.348840 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-home\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.349061 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-model-cache" (OuterVolumeSpecName: "model-cache") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:25.349061 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbc904-4cca-442e-b2ea-337847a0a119-tls-certs\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.349061 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.348887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-tmp-dir\") pod \"04bbc904-4cca-442e-b2ea-337847a0a119\" (UID: \"04bbc904-4cca-442e-b2ea-337847a0a119\") "
Sep 17 18:43:25.349331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.349203    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.349630 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.349606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-home" (OuterVolumeSpecName: "home") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:25.349788 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.349697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bbc904-4cca-442e-b2ea-337847a0a119-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:25.351678 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.351656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-dshm" (OuterVolumeSpecName: "dshm") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:25.351905 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.351877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bbc904-4cca-442e-b2ea-337847a0a119-kube-api-access-cw5zz" (OuterVolumeSpecName: "kube-api-access-cw5zz") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "kube-api-access-cw5zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:25.352275 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.352249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bbc904-4cca-442e-b2ea-337847a0a119-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:25.368811 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.368777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:25.375572 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.375548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04bbc904-4cca-442e-b2ea-337847a0a119" (UID: "04bbc904-4cca-442e-b2ea-337847a0a119"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:25.449700 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.449675    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw5zz\" (UniqueName: \"kubernetes.io/projected/04bbc904-4cca-442e-b2ea-337847a0a119-kube-api-access-cw5zz\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.449700 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.449701    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.449824 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.449712    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bbc904-4cca-442e-b2ea-337847a0a119-cabundle-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.449824 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.449723    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-kserve-provision-location\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.449824 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.449732    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.449824 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.449742    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbc904-4cca-442e-b2ea-337847a0a119-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.449824 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.449752    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbc904-4cca-442e-b2ea-337847a0a119-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.697770 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.697740    2580 generic.go:356] "Generic (PLEG): container finished" podID="04bbc904-4cca-442e-b2ea-337847a0a119" containerID="0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66" exitCode=0
Sep 17 18:43:25.698164 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.697790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" event={"ID":"04bbc904-4cca-442e-b2ea-337847a0a119","Type":"ContainerDied","Data":"0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66"}
Sep 17 18:43:25.698164 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.697830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" event={"ID":"04bbc904-4cca-442e-b2ea-337847a0a119","Type":"ContainerDied","Data":"9b9d22a22ac879359156829bc1925d95f68752cdf12bc3f8f337dd5df53bc7fe"}
Sep 17 18:43:25.698164 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.697851    2580 scope.go:117] "RemoveContainer" containerID="0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66"
Sep 17 18:43:25.698164 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.697793    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"
Sep 17 18:43:25.707848 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.707832    2580 scope.go:117] "RemoveContainer" containerID="f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84"
Sep 17 18:43:25.715621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.715604    2580 scope.go:117] "RemoveContainer" containerID="0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66"
Sep 17 18:43:25.715877 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:43:25.715855    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66\": container with ID starting with 0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66 not found: ID does not exist" containerID="0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66"
Sep 17 18:43:25.715929 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.715885    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66"} err="failed to get container status \"0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66\": rpc error: code = NotFound desc = could not find container \"0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66\": container with ID starting with 0dc77bc670a3a9befad3e189afa6329dedde7bfc2f6c6cd7cb05365cfb409b66 not found: ID does not exist"
Sep 17 18:43:25.715929 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.715902    2580 scope.go:117] "RemoveContainer" containerID="f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84"
Sep 17 18:43:25.724220 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:43:25.724187    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84\": container with ID starting with f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84 not found: ID does not exist" containerID="f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84"
Sep 17 18:43:25.724321 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.724246    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84"} err="failed to get container status \"f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84\": rpc error: code = NotFound desc = could not find container \"f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84\": container with ID starting with f274ca6458e6967f5a4539fd1d742d342bd1492b83850628003123f926d5fa84 not found: ID does not exist"
Sep 17 18:43:25.727144 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.727120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"]
Sep 17 18:43:25.729165 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:25.729145    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd"]
Sep 17 18:43:26.046410 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:26.046329    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" path="/var/lib/kubelet/pods/04bbc904-4cca-442e-b2ea-337847a0a119/volumes"
Sep 17 18:43:26.235033 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:26.234995    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78dbc984b4-vqtfd" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:43:27.824577 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:27.824539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:27.880683 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:27.880633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:40.235078 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.235044    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"]
Sep 17 18:43:40.235616 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.235033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:40.245429 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:43:40.245385    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:43:40.245429 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:43:40.245429 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:43:40.245429 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0" cmd=["/bin/sleep","15"]
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.245453    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" message="" err=<
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:43:40.245469    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" podUID="242a7208-73fa-4d72-a34d-422d344609f4" containerName="main" containerID="cri-o://050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0"
Sep 17 18:43:40.245729 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.245523    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" podUID="242a7208-73fa-4d72-a34d-422d344609f4" containerName="main" containerID="cri-o://050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0" gracePeriod=60
Sep 17 18:43:40.326599 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.326566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:40.599153 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.599129    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:40.673847 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.673819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-home\") pod \"242a7208-73fa-4d72-a34d-422d344609f4\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") "
Sep 17 18:43:40.674027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.673871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242a7208-73fa-4d72-a34d-422d344609f4-tls-certs\") pod \"242a7208-73fa-4d72-a34d-422d344609f4\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") "
Sep 17 18:43:40.674027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.673897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-tmp-dir\") pod \"242a7208-73fa-4d72-a34d-422d344609f4\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") "
Sep 17 18:43:40.674027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.673916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-dshm\") pod \"242a7208-73fa-4d72-a34d-422d344609f4\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") "
Sep 17 18:43:40.674027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.673943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw828\" (UniqueName: \"kubernetes.io/projected/242a7208-73fa-4d72-a34d-422d344609f4-kube-api-access-nw828\") pod \"242a7208-73fa-4d72-a34d-422d344609f4\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") "
Sep 17 18:43:40.674027 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.674005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-model-cache\") pod \"242a7208-73fa-4d72-a34d-422d344609f4\" (UID: \"242a7208-73fa-4d72-a34d-422d344609f4\") "
Sep 17 18:43:40.674342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.674150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-home" (OuterVolumeSpecName: "home") pod "242a7208-73fa-4d72-a34d-422d344609f4" (UID: "242a7208-73fa-4d72-a34d-422d344609f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:40.674342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.674160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "242a7208-73fa-4d72-a34d-422d344609f4" (UID: "242a7208-73fa-4d72-a34d-422d344609f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:40.674342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.674258    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:40.674342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.674278    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:40.674510 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.674379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "242a7208-73fa-4d72-a34d-422d344609f4" (UID: "242a7208-73fa-4d72-a34d-422d344609f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:40.676276 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.676247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242a7208-73fa-4d72-a34d-422d344609f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "242a7208-73fa-4d72-a34d-422d344609f4" (UID: "242a7208-73fa-4d72-a34d-422d344609f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:40.676381 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.676267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242a7208-73fa-4d72-a34d-422d344609f4-kube-api-access-nw828" (OuterVolumeSpecName: "kube-api-access-nw828") pod "242a7208-73fa-4d72-a34d-422d344609f4" (UID: "242a7208-73fa-4d72-a34d-422d344609f4"). InnerVolumeSpecName "kube-api-access-nw828". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:40.676381 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.676280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-dshm" (OuterVolumeSpecName: "dshm") pod "242a7208-73fa-4d72-a34d-422d344609f4" (UID: "242a7208-73fa-4d72-a34d-422d344609f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:40.775254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.775197    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:40.775254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.775217    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw828\" (UniqueName: \"kubernetes.io/projected/242a7208-73fa-4d72-a34d-422d344609f4-kube-api-access-nw828\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:40.775254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.775229    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242a7208-73fa-4d72-a34d-422d344609f4-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:40.775254 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.775238    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242a7208-73fa-4d72-a34d-422d344609f4-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:43:40.780338 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.780311    2580 generic.go:356] "Generic (PLEG): container finished" podID="242a7208-73fa-4d72-a34d-422d344609f4" containerID="050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0" exitCode=0
Sep 17 18:43:40.780440 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.780359    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"
Sep 17 18:43:40.780440 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.780406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" event={"ID":"242a7208-73fa-4d72-a34d-422d344609f4","Type":"ContainerDied","Data":"050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0"}
Sep 17 18:43:40.780561 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.780467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7" event={"ID":"242a7208-73fa-4d72-a34d-422d344609f4","Type":"ContainerDied","Data":"dfc6eeaf8dc4c440829a8549ebd102813e75b1fea3eea5a4822c3b58441180b0"}
Sep 17 18:43:40.780561 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.780501    2580 scope.go:117] "RemoveContainer" containerID="050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0"
Sep 17 18:43:40.794401 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.794379    2580 scope.go:117] "RemoveContainer" containerID="050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0"
Sep 17 18:43:40.794671 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:43:40.794651    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0\": container with ID starting with 050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0 not found: ID does not exist" containerID="050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0"
Sep 17 18:43:40.794733 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.794680    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0"} err="failed to get container status \"050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0\": rpc error: code = NotFound desc = could not find container \"050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0\": container with ID starting with 050cdbc6e55e3c602e00ac715a97358bb1eace8221a602f558718e84d892bfd0 not found: ID does not exist"
Sep 17 18:43:40.804249 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.804223    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"]
Sep 17 18:43:40.808708 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:40.808687    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jczh7"]
Sep 17 18:43:42.045472 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:42.045440    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242a7208-73fa-4d72-a34d-422d344609f4" path="/var/lib/kubelet/pods/242a7208-73fa-4d72-a34d-422d344609f4/volumes"
Sep 17 18:43:57.128221 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:43:57.128158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:19.815262 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:19.815206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:30.308235 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:30.308194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:33.023566 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:33.023522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:33.111821 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:33.111780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:36.539591 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:36.539529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:39.453206 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:39.453169    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:42.149483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:42.149445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:42.210245 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:42.210188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:45.219531 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:45.219447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:45.319876 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:45.319834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.391927 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.391889    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"]
Sep 17 18:44:49.392331 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392316    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="storage-initializer"
Sep 17 18:44:49.392376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392332    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="storage-initializer"
Sep 17 18:44:49.392376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392341    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main"
Sep 17 18:44:49.392376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main"
Sep 17 18:44:49.392376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392355    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242a7208-73fa-4d72-a34d-422d344609f4" containerName="main"
Sep 17 18:44:49.392376 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392360    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="242a7208-73fa-4d72-a34d-422d344609f4" containerName="main"
Sep 17 18:44:49.392543 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392434    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bbc904-4cca-442e-b2ea-337847a0a119" containerName="main"
Sep 17 18:44:49.392543 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.392442    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="242a7208-73fa-4d72-a34d-422d344609f4" containerName="main"
Sep 17 18:44:49.395548 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.395530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.398270 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.398250    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 18:44:49.398367 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.398252    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 18:44:49.399305 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.399283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 18:44:49.399305 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.399300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-xhz7f\""
Sep 17 18:44:49.405217 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.405191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"]
Sep 17 18:44:49.420468 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.420435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.517166 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.517129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.517328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.517185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.517328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.517223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.517328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.517286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgnlb\" (UniqueName: \"kubernetes.io/projected/565b7808-72f2-4bec-8660-90406d1ef7d4-kube-api-access-cgnlb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.517328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.517316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565b7808-72f2-4bec-8660-90406d1ef7d4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.517512 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.517377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.617420 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.617381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.617750 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.617728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565b7808-72f2-4bec-8660-90406d1ef7d4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.617808 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.617773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.617848 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.617806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.617848 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.617840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.617923 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.617880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.617923 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.617909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgnlb\" (UniqueName: \"kubernetes.io/projected/565b7808-72f2-4bec-8660-90406d1ef7d4-kube-api-access-cgnlb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.618367 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.618343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.618491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.618393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.618491 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.618410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.620216 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.620193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.620579 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.620563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565b7808-72f2-4bec-8660-90406d1ef7d4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.625799 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.625772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgnlb\" (UniqueName: \"kubernetes.io/projected/565b7808-72f2-4bec-8660-90406d1ef7d4-kube-api-access-cgnlb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-fwjs5\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.706930 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.706845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:49.985192 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:49.982392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"]
Sep 17 18:44:49.985965 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:44:49.985927    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565b7808_72f2_4bec_8660_90406d1ef7d4.slice/crio-776fadbaeb8159b228ce706613bb366d75ff7b0d8c7585d389ca515e355034f5 WatchSource:0}: Error finding container 776fadbaeb8159b228ce706613bb366d75ff7b0d8c7585d389ca515e355034f5: Status 404 returned error can't find the container with id 776fadbaeb8159b228ce706613bb366d75ff7b0d8c7585d389ca515e355034f5
Sep 17 18:44:50.150621 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:50.150577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" event={"ID":"565b7808-72f2-4bec-8660-90406d1ef7d4","Type":"ContainerStarted","Data":"32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52"}
Sep 17 18:44:50.150826 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:50.150649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" event={"ID":"565b7808-72f2-4bec-8660-90406d1ef7d4","Type":"ContainerStarted","Data":"776fadbaeb8159b228ce706613bb366d75ff7b0d8c7585d389ca515e355034f5"}
Sep 17 18:44:50.176776 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:50.176718    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" podStartSLOduration=1.176701862 podStartE2EDuration="1.176701862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:50.174419192 +0000 UTC m=+3866.513812783" watchObservedRunningTime="2026-09-17 18:44:50.176701862 +0000 UTC m=+3866.516095452"
Sep 17 18:44:50.707946 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:50.707899    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:59.708019 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:59.707974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:44:59.721228 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:44:59.721194    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:45:00.223974 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:00.223941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:45:10.668356 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:10.668306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:22.962743 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:22.962696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:23.062972 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.062933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:23.617391 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.617349    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"]
Sep 17 18:45:23.620519 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.620497    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.632524 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.632500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"]
Sep 17 18:45:23.715647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.715610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7d2\" (UniqueName: \"kubernetes.io/projected/6cfabae3-4ce9-4713-a94f-47cd18fdedda-kube-api-access-2g7d2\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.715838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.715662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.715838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.715720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.715838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.715806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.715838 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.715830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.716021 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.715861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817207 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817207 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817452 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817452 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7d2\" (UniqueName: \"kubernetes.io/projected/6cfabae3-4ce9-4713-a94f-47cd18fdedda-kube-api-access-2g7d2\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817452 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817452 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817702 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817755 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.817825 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.817803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.819584 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.819564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.819880 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.819861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.825573 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.825548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7d2\" (UniqueName: \"kubernetes.io/projected/6cfabae3-4ce9-4713-a94f-47cd18fdedda-kube-api-access-2g7d2\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-cbclg\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:23.932256 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:23.932218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:24.215121 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:24.215065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"]
Sep 17 18:45:24.216764 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:45:24.216726    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfabae3_4ce9_4713_a94f_47cd18fdedda.slice/crio-42f3fca93b5d583c9adc6bb9939a6e7ed86e1a82885156285cead220ec7a77d7 WatchSource:0}: Error finding container 42f3fca93b5d583c9adc6bb9939a6e7ed86e1a82885156285cead220ec7a77d7: Status 404 returned error can't find the container with id 42f3fca93b5d583c9adc6bb9939a6e7ed86e1a82885156285cead220ec7a77d7
Sep 17 18:45:24.340467 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:24.340427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" event={"ID":"6cfabae3-4ce9-4713-a94f-47cd18fdedda","Type":"ContainerStarted","Data":"8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0"}
Sep 17 18:45:24.340655 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:24.340488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" event={"ID":"6cfabae3-4ce9-4713-a94f-47cd18fdedda","Type":"ContainerStarted","Data":"42f3fca93b5d583c9adc6bb9939a6e7ed86e1a82885156285cead220ec7a77d7"}
Sep 17 18:45:24.359147 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:24.359069    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" podStartSLOduration=1.359049263 podStartE2EDuration="1.359049263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:24.35701903 +0000 UTC m=+3900.696412620" watchObservedRunningTime="2026-09-17 18:45:24.359049263 +0000 UTC m=+3900.698442864"
Sep 17 18:45:24.932547 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:24.932512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:25.479001 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:25.478972    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:45:25.495642 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:25.495612    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:45:32.301533 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.301492    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"]
Sep 17 18:45:32.305148 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.305114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"]
Sep 17 18:45:32.312904 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:45:32.312864    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:45:32.312904 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:32.312904 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:32.312904 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0" cmd=["/bin/sleep","15"]
Sep 17 18:45:32.313289 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.313264    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" message="" err=<
Sep 17 18:45:32.313289 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:32.313289 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:32.313289 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:45:32.313432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.313167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:32.313432 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:45:32.313291    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:45:32.313432 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:32.313432 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:32.313432 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" podUID="6cfabae3-4ce9-4713-a94f-47cd18fdedda" containerName="main" containerID="cri-o://8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0"
Sep 17 18:45:32.313623 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.313415    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" podUID="6cfabae3-4ce9-4713-a94f-47cd18fdedda" containerName="main" containerID="cri-o://8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0" gracePeriod=60
Sep 17 18:45:32.314966 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:45:32.314929    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:45:32.314966 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:32.314966 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:32.314966 ip-10-0-139-39 kubenswrapper[2580]:  > containerID="32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52" cmd=["/bin/sleep","15"]
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.314981    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" message="" err=<
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]:  >
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:45:32.314997    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" podUID="565b7808-72f2-4bec-8660-90406d1ef7d4" containerName="main" containerID="cri-o://32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52"
Sep 17 18:45:32.315243 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.315037    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" podUID="565b7808-72f2-4bec-8660-90406d1ef7d4" containerName="main" containerID="cri-o://32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52" gracePeriod=60
Sep 17 18:45:32.415017 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.414977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:32.695680 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.695653    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:45:32.699746 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.699719    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:32.792495 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-dshm\") pod \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") "
Sep 17 18:45:32.792495 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792501    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-tmp-dir\") pod \"565b7808-72f2-4bec-8660-90406d1ef7d4\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") "
Sep 17 18:45:32.792745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-home\") pod \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") "
Sep 17 18:45:32.792745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792579    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-model-cache\") pod \"565b7808-72f2-4bec-8660-90406d1ef7d4\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") "
Sep 17 18:45:32.792745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tls-certs\") pod \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") "
Sep 17 18:45:32.792745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g7d2\" (UniqueName: \"kubernetes.io/projected/6cfabae3-4ce9-4713-a94f-47cd18fdedda-kube-api-access-2g7d2\") pod \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") "
Sep 17 18:45:32.792745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgnlb\" (UniqueName: \"kubernetes.io/projected/565b7808-72f2-4bec-8660-90406d1ef7d4-kube-api-access-cgnlb\") pod \"565b7808-72f2-4bec-8660-90406d1ef7d4\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") "
Sep 17 18:45:32.792745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-home\") pod \"565b7808-72f2-4bec-8660-90406d1ef7d4\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") "
Sep 17 18:45:32.792745 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-model-cache\") pod \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") "
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792775    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565b7808-72f2-4bec-8660-90406d1ef7d4-tls-certs\") pod \"565b7808-72f2-4bec-8660-90406d1ef7d4\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") "
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tmp-dir\") pod \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\" (UID: \"6cfabae3-4ce9-4713-a94f-47cd18fdedda\") "
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-dshm\") pod \"565b7808-72f2-4bec-8660-90406d1ef7d4\" (UID: \"565b7808-72f2-4bec-8660-90406d1ef7d4\") "
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "565b7808-72f2-4bec-8660-90406d1ef7d4" (UID: "565b7808-72f2-4bec-8660-90406d1ef7d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-home" (OuterVolumeSpecName: "home") pod "6cfabae3-4ce9-4713-a94f-47cd18fdedda" (UID: "6cfabae3-4ce9-4713-a94f-47cd18fdedda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.792979    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-home" (OuterVolumeSpecName: "home") pod "565b7808-72f2-4bec-8660-90406d1ef7d4" (UID: "565b7808-72f2-4bec-8660-90406d1ef7d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.793108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cfabae3-4ce9-4713-a94f-47cd18fdedda" (UID: "6cfabae3-4ce9-4713-a94f-47cd18fdedda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.793136 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.793114    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.793556 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.793174    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.793556 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.793183    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cfabae3-4ce9-4713-a94f-47cd18fdedda" (UID: "6cfabae3-4ce9-4713-a94f-47cd18fdedda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.793556 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.793192    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.793556 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.793255    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "565b7808-72f2-4bec-8660-90406d1ef7d4" (UID: "565b7808-72f2-4bec-8660-90406d1ef7d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.795068 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.795043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-dshm" (OuterVolumeSpecName: "dshm") pod "6cfabae3-4ce9-4713-a94f-47cd18fdedda" (UID: "6cfabae3-4ce9-4713-a94f-47cd18fdedda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.795262 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.795240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-dshm" (OuterVolumeSpecName: "dshm") pod "565b7808-72f2-4bec-8660-90406d1ef7d4" (UID: "565b7808-72f2-4bec-8660-90406d1ef7d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:32.795480 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.795459    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565b7808-72f2-4bec-8660-90406d1ef7d4-kube-api-access-cgnlb" (OuterVolumeSpecName: "kube-api-access-cgnlb") pod "565b7808-72f2-4bec-8660-90406d1ef7d4" (UID: "565b7808-72f2-4bec-8660-90406d1ef7d4"). InnerVolumeSpecName "kube-api-access-cgnlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:32.795554 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.795508    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565b7808-72f2-4bec-8660-90406d1ef7d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "565b7808-72f2-4bec-8660-90406d1ef7d4" (UID: "565b7808-72f2-4bec-8660-90406d1ef7d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:32.795597 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.795575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cfabae3-4ce9-4713-a94f-47cd18fdedda" (UID: "6cfabae3-4ce9-4713-a94f-47cd18fdedda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:32.795634 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.795598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfabae3-4ce9-4713-a94f-47cd18fdedda-kube-api-access-2g7d2" (OuterVolumeSpecName: "kube-api-access-2g7d2") pod "6cfabae3-4ce9-4713-a94f-47cd18fdedda" (UID: "6cfabae3-4ce9-4713-a94f-47cd18fdedda"). InnerVolumeSpecName "kube-api-access-2g7d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:32.894514 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894481    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7d2\" (UniqueName: \"kubernetes.io/projected/6cfabae3-4ce9-4713-a94f-47cd18fdedda-kube-api-access-2g7d2\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894518    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgnlb\" (UniqueName: \"kubernetes.io/projected/565b7808-72f2-4bec-8660-90406d1ef7d4-kube-api-access-cgnlb\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894534    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894549    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565b7808-72f2-4bec-8660-90406d1ef7d4-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894563    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894574    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894586    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfabae3-4ce9-4713-a94f-47cd18fdedda-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894596    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565b7808-72f2-4bec-8660-90406d1ef7d4-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:32.894670 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:32.894607    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfabae3-4ce9-4713-a94f-47cd18fdedda-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:45:33.393445 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.393407    2580 generic.go:356] "Generic (PLEG): container finished" podID="565b7808-72f2-4bec-8660-90406d1ef7d4" containerID="32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52" exitCode=0
Sep 17 18:45:33.393892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.393455    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"
Sep 17 18:45:33.393892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.393493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" event={"ID":"565b7808-72f2-4bec-8660-90406d1ef7d4","Type":"ContainerDied","Data":"32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52"}
Sep 17 18:45:33.393892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.393546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5" event={"ID":"565b7808-72f2-4bec-8660-90406d1ef7d4","Type":"ContainerDied","Data":"776fadbaeb8159b228ce706613bb366d75ff7b0d8c7585d389ca515e355034f5"}
Sep 17 18:45:33.393892 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.393575    2580 scope.go:117] "RemoveContainer" containerID="32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52"
Sep 17 18:45:33.395018 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.394884    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cfabae3-4ce9-4713-a94f-47cd18fdedda" containerID="8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0" exitCode=0
Sep 17 18:45:33.395018 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.394917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" event={"ID":"6cfabae3-4ce9-4713-a94f-47cd18fdedda","Type":"ContainerDied","Data":"8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0"}
Sep 17 18:45:33.395018 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.394929    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"
Sep 17 18:45:33.395018 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.394958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg" event={"ID":"6cfabae3-4ce9-4713-a94f-47cd18fdedda","Type":"ContainerDied","Data":"42f3fca93b5d583c9adc6bb9939a6e7ed86e1a82885156285cead220ec7a77d7"}
Sep 17 18:45:33.403668 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.403645    2580 scope.go:117] "RemoveContainer" containerID="32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52"
Sep 17 18:45:33.403971 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:45:33.403952    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52\": container with ID starting with 32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52 not found: ID does not exist" containerID="32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52"
Sep 17 18:45:33.404035 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.403980    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52"} err="failed to get container status \"32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52\": rpc error: code = NotFound desc = could not find container \"32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52\": container with ID starting with 32ccb53c1f7ce4499dfff029e5fb1766b54e1eb1ac852f75e1aa3e3e84a48b52 not found: ID does not exist"
Sep 17 18:45:33.404035 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.403997    2580 scope.go:117] "RemoveContainer" containerID="8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0"
Sep 17 18:45:33.412981 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.412961    2580 scope.go:117] "RemoveContainer" containerID="8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0"
Sep 17 18:45:33.413259 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:45:33.413240    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0\": container with ID starting with 8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0 not found: ID does not exist" containerID="8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0"
Sep 17 18:45:33.413318 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.413265    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0"} err="failed to get container status \"8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0\": rpc error: code = NotFound desc = could not find container \"8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0\": container with ID starting with 8bfa398808fbb91095b73cfc5d3ceeb19c72825c7643d151a2c6d07ed167a4d0 not found: ID does not exist"
Sep 17 18:45:33.425131 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.423192    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"]
Sep 17 18:45:33.430287 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.430264    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-fwjs5"]
Sep 17 18:45:33.442618 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.442598    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"]
Sep 17 18:45:33.446763 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:33.446741    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-cbclg"]
Sep 17 18:45:34.045815 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:34.045775    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565b7808-72f2-4bec-8660-90406d1ef7d4" path="/var/lib/kubelet/pods/565b7808-72f2-4bec-8660-90406d1ef7d4/volumes"
Sep 17 18:45:34.046316 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:45:34.046293    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cfabae3-4ce9-4713-a94f-47cd18fdedda" path="/var/lib/kubelet/pods/6cfabae3-4ce9-4713-a94f-47cd18fdedda/volumes"
Sep 17 18:46:20.688067 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:20.688012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:22.264550 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.264503    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v642p/must-gather-7tfc9"]
Sep 17 18:46:22.265112 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.265073    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565b7808-72f2-4bec-8660-90406d1ef7d4" containerName="main"
Sep 17 18:46:22.265201 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.265114    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="565b7808-72f2-4bec-8660-90406d1ef7d4" containerName="main"
Sep 17 18:46:22.265201 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.265140    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cfabae3-4ce9-4713-a94f-47cd18fdedda" containerName="main"
Sep 17 18:46:22.265201 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.265150    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cfabae3-4ce9-4713-a94f-47cd18fdedda" containerName="main"
Sep 17 18:46:22.265365 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.265245    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="565b7808-72f2-4bec-8660-90406d1ef7d4" containerName="main"
Sep 17 18:46:22.265365 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.265260    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cfabae3-4ce9-4713-a94f-47cd18fdedda" containerName="main"
Sep 17 18:46:22.268488 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.268461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.271236 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.271209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v642p\"/\"openshift-service-ca.crt\""
Sep 17 18:46:22.271236 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.271221    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v642p\"/\"kube-root-ca.crt\""
Sep 17 18:46:22.271462 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.271366    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v642p\"/\"default-dockercfg-zmvrl\""
Sep 17 18:46:22.277432 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.277404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v642p/must-gather-7tfc9"]
Sep 17 18:46:22.288757 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.288721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:22.311518 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.311477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmmc\" (UniqueName: \"kubernetes.io/projected/205c2c8d-202a-41c8-8e1a-fb99f184298b-kube-api-access-gsmmc\") pod \"must-gather-7tfc9\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") " pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.311727 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.311600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/205c2c8d-202a-41c8-8e1a-fb99f184298b-must-gather-output\") pod \"must-gather-7tfc9\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") " pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.358645 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.358599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:22.412480 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.412433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/205c2c8d-202a-41c8-8e1a-fb99f184298b-must-gather-output\") pod \"must-gather-7tfc9\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") " pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.412686 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.412498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmmc\" (UniqueName: \"kubernetes.io/projected/205c2c8d-202a-41c8-8e1a-fb99f184298b-kube-api-access-gsmmc\") pod \"must-gather-7tfc9\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") " pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.412769 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.412754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/205c2c8d-202a-41c8-8e1a-fb99f184298b-must-gather-output\") pod \"must-gather-7tfc9\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") " pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.421541 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.421505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmmc\" (UniqueName: \"kubernetes.io/projected/205c2c8d-202a-41c8-8e1a-fb99f184298b-kube-api-access-gsmmc\") pod \"must-gather-7tfc9\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") " pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.596642 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.596550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:22.872900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.872868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v642p/must-gather-7tfc9"]
Sep 17 18:46:22.875461 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:46:22.875427    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod205c2c8d_202a_41c8_8e1a_fb99f184298b.slice/crio-5561a5472760ec173a0a3272d53e735955a505ee76085aae9f9f75e73732fc63 WatchSource:0}: Error finding container 5561a5472760ec173a0a3272d53e735955a505ee76085aae9f9f75e73732fc63: Status 404 returned error can't find the container with id 5561a5472760ec173a0a3272d53e735955a505ee76085aae9f9f75e73732fc63
Sep 17 18:46:22.877828 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:22.877806    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:46:23.676263 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:23.676206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v642p/must-gather-7tfc9" event={"ID":"205c2c8d-202a-41c8-8e1a-fb99f184298b","Type":"ContainerStarted","Data":"5561a5472760ec173a0a3272d53e735955a505ee76085aae9f9f75e73732fc63"}
Sep 17 18:46:27.206442 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:27.206394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:29.718266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:29.718150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v642p/must-gather-7tfc9" event={"ID":"205c2c8d-202a-41c8-8e1a-fb99f184298b","Type":"ContainerStarted","Data":"09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d"}
Sep 17 18:46:29.718266 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:29.718196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v642p/must-gather-7tfc9" event={"ID":"205c2c8d-202a-41c8-8e1a-fb99f184298b","Type":"ContainerStarted","Data":"2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd"}
Sep 17 18:46:29.734583 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:29.734466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v642p/must-gather-7tfc9" podStartSLOduration=1.640028671 podStartE2EDuration="7.734443215s" totalImagesPullingTime="6.094414544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:29.733413962 +0000 UTC m=+3966.072807578" watchObservedRunningTime="2026-09-17 18:46:29.734443215 +0000 UTC m=+3966.073836808"
Sep 17 18:46:33.216976 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:33.216926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:33.317562 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:33.317519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:46.468797 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:46.468750    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79d7b547b8-gfldz_63f2f4e6-aba2-4b69-8ef9-f2b68013d13e/router/0.log"
Sep 17 18:46:47.281185 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:47.281136    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79d7b547b8-gfldz_63f2f4e6-aba2-4b69-8ef9-f2b68013d13e/router/0.log"
Sep 17 18:46:48.544680 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:48.544639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-rk659_b6cef6a2-dfb4-4399-9b2f-0ea208409579/manager/0.log"
Sep 17 18:46:48.553805 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:48.553775    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wzhjg_16466b21-0f3d-49c6-abef-4d4c9c15a7ee/limitador/0.log"
Sep 17 18:46:48.571479 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:48.571433    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-k7bhn_c6db2741-11b5-4284-886b-a6829b728d09/manager/0.log"
Sep 17 18:46:53.883860 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:53.883828    2580 generic.go:356] "Generic (PLEG): container finished" podID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerID="2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd" exitCode=0
Sep 17 18:46:53.884328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:53.883900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v642p/must-gather-7tfc9" event={"ID":"205c2c8d-202a-41c8-8e1a-fb99f184298b","Type":"ContainerDied","Data":"2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd"}
Sep 17 18:46:53.884328 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:53.884308    2580 scope.go:117] "RemoveContainer" containerID="2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd"
Sep 17 18:46:53.929131 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:53.929076    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v642p_must-gather-7tfc9_205c2c8d-202a-41c8-8e1a-fb99f184298b/gather/0.log"
Sep 17 18:46:54.345483 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.345448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:54.471996 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.471951    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rx6xt/must-gather-cc6bg"]
Sep 17 18:46:54.476674 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.476647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:54.479290 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.479265    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rx6xt\"/\"default-dockercfg-lm5ng\""
Sep 17 18:46:54.479429 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.479284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rx6xt\"/\"kube-root-ca.crt\""
Sep 17 18:46:54.479429 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.479265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rx6xt\"/\"openshift-service-ca.crt\""
Sep 17 18:46:54.488004 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.487968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rx6xt/must-gather-cc6bg"]
Sep 17 18:46:54.491729 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.491695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:54.592610 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.592567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:54.609252 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.609162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/950fcc7d-344d-462e-9b3b-bcb4274c5fa3-must-gather-output\") pod \"must-gather-cc6bg\" (UID: \"950fcc7d-344d-462e-9b3b-bcb4274c5fa3\") " pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:54.609408 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.609260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzq7\" (UniqueName: \"kubernetes.io/projected/950fcc7d-344d-462e-9b3b-bcb4274c5fa3-kube-api-access-kwzq7\") pod \"must-gather-cc6bg\" (UID: \"950fcc7d-344d-462e-9b3b-bcb4274c5fa3\") " pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:54.710418 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.710373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzq7\" (UniqueName: \"kubernetes.io/projected/950fcc7d-344d-462e-9b3b-bcb4274c5fa3-kube-api-access-kwzq7\") pod \"must-gather-cc6bg\" (UID: \"950fcc7d-344d-462e-9b3b-bcb4274c5fa3\") " pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:54.710418 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.710430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/950fcc7d-344d-462e-9b3b-bcb4274c5fa3-must-gather-output\") pod \"must-gather-cc6bg\" (UID: \"950fcc7d-344d-462e-9b3b-bcb4274c5fa3\") " pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:54.710871 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.710848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/950fcc7d-344d-462e-9b3b-bcb4274c5fa3-must-gather-output\") pod \"must-gather-cc6bg\" (UID: \"950fcc7d-344d-462e-9b3b-bcb4274c5fa3\") " pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:54.718561 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.718534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzq7\" (UniqueName: \"kubernetes.io/projected/950fcc7d-344d-462e-9b3b-bcb4274c5fa3-kube-api-access-kwzq7\") pod \"must-gather-cc6bg\" (UID: \"950fcc7d-344d-462e-9b3b-bcb4274c5fa3\") " pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:54.787746 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:54.787702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rx6xt/must-gather-cc6bg"
Sep 17 18:46:55.061893 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:55.061862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rx6xt/must-gather-cc6bg"]
Sep 17 18:46:55.064072 ip-10-0-139-39 kubenswrapper[2580]: W0917 18:46:55.064045    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod950fcc7d_344d_462e_9b3b_bcb4274c5fa3.slice/crio-1bc1a732b22ed4255accd02c9cb55fe6d1ed25fbbb7ca29448ccf81d44978ae3 WatchSource:0}: Error finding container 1bc1a732b22ed4255accd02c9cb55fe6d1ed25fbbb7ca29448ccf81d44978ae3: Status 404 returned error can't find the container with id 1bc1a732b22ed4255accd02c9cb55fe6d1ed25fbbb7ca29448ccf81d44978ae3
Sep 17 18:46:55.896729 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:55.896699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rx6xt/must-gather-cc6bg" event={"ID":"950fcc7d-344d-462e-9b3b-bcb4274c5fa3","Type":"ContainerStarted","Data":"9be8c5b1c4f090e8c67915bcb33b054bd378cc47523aa857709214d76b738b07"}
Sep 17 18:46:55.897045 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:55.896743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rx6xt/must-gather-cc6bg" event={"ID":"950fcc7d-344d-462e-9b3b-bcb4274c5fa3","Type":"ContainerStarted","Data":"1bc1a732b22ed4255accd02c9cb55fe6d1ed25fbbb7ca29448ccf81d44978ae3"}
Sep 17 18:46:56.907819 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:56.907782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rx6xt/must-gather-cc6bg" event={"ID":"950fcc7d-344d-462e-9b3b-bcb4274c5fa3","Type":"ContainerStarted","Data":"e888ddcfc25f1bbbfa1ec0263362582d821a75fef114f55dda49bcbe5d4d240c"}
Sep 17 18:46:56.924718 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:56.924659    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rx6xt/must-gather-cc6bg" podStartSLOduration=2.220489374 podStartE2EDuration="2.924640456s" totalImagesPullingTime="704.151082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:56.923018278 +0000 UTC m=+3993.262411882" watchObservedRunningTime="2026-09-17 18:46:56.924640456 +0000 UTC m=+3993.264034086"
Sep 17 18:46:57.451998 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:57.451963    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2tnhn_45cd1091-7a05-4050-9ea5-fc2c920202bf/global-pull-secret-syncer/0.log"
Sep 17 18:46:57.569444 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:57.569412    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log"
Sep 17 18:46:57.707901 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:57.707812    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-39.ec2.internal_46cd3fb458d38324dc66491843cdea97/haproxy/0.log"
Sep 17 18:46:59.330504 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.330461    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-v642p/must-gather-7tfc9"]
Sep 17 18:46:59.331523 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.331452    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-v642p/must-gather-7tfc9" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerName="copy" containerID="cri-o://09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d" gracePeriod=2
Sep 17 18:46:59.333821 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:46:59.333730    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7tfc9\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v642p\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:59.335882 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.335843    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-v642p/must-gather-7tfc9"]
Sep 17 18:46:59.716694 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.716606    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v642p_must-gather-7tfc9_205c2c8d-202a-41c8-8e1a-fb99f184298b/copy/0.log"
Sep 17 18:46:59.721470 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.721391    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:59.724994 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:46:59.724380    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7tfc9\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v642p\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:59.865282 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.865239    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/205c2c8d-202a-41c8-8e1a-fb99f184298b-must-gather-output\") pod \"205c2c8d-202a-41c8-8e1a-fb99f184298b\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") "
Sep 17 18:46:59.865636 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.865613    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsmmc\" (UniqueName: \"kubernetes.io/projected/205c2c8d-202a-41c8-8e1a-fb99f184298b-kube-api-access-gsmmc\") pod \"205c2c8d-202a-41c8-8e1a-fb99f184298b\" (UID: \"205c2c8d-202a-41c8-8e1a-fb99f184298b\") "
Sep 17 18:46:59.874726 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.874664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205c2c8d-202a-41c8-8e1a-fb99f184298b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "205c2c8d-202a-41c8-8e1a-fb99f184298b" (UID: "205c2c8d-202a-41c8-8e1a-fb99f184298b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:59.888098 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.888023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205c2c8d-202a-41c8-8e1a-fb99f184298b-kube-api-access-gsmmc" (OuterVolumeSpecName: "kube-api-access-gsmmc") pod "205c2c8d-202a-41c8-8e1a-fb99f184298b" (UID: "205c2c8d-202a-41c8-8e1a-fb99f184298b"). InnerVolumeSpecName "kube-api-access-gsmmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:59.931548 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.931462    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v642p_must-gather-7tfc9_205c2c8d-202a-41c8-8e1a-fb99f184298b/copy/0.log"
Sep 17 18:46:59.931960 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.931926    2580 generic.go:356] "Generic (PLEG): container finished" podID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerID="09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d" exitCode=143
Sep 17 18:46:59.932181 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.932158    2580 scope.go:117] "RemoveContainer" containerID="09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d"
Sep 17 18:46:59.932360 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.932342    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:59.956162 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:46:59.955603    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7tfc9\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v642p\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:46:59.971126 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.970589    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/205c2c8d-202a-41c8-8e1a-fb99f184298b-must-gather-output\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:46:59.971126 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.970625    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsmmc\" (UniqueName: \"kubernetes.io/projected/205c2c8d-202a-41c8-8e1a-fb99f184298b-kube-api-access-gsmmc\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\""
Sep 17 18:46:59.971126 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:46:59.970785    2580 scope.go:117] "RemoveContainer" containerID="2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd"
Sep 17 18:46:59.994011 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:46:59.993946    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7tfc9\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v642p\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" pod="openshift-must-gather-v642p/must-gather-7tfc9"
Sep 17 18:47:00.019371 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:00.019231    2580 scope.go:117] "RemoveContainer" containerID="09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d"
Sep 17 18:47:00.020664 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:47:00.020510    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d\": container with ID starting with 09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d not found: ID does not exist" containerID="09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d"
Sep 17 18:47:00.020664 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:00.020550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d"} err="failed to get container status \"09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d\": rpc error: code = NotFound desc = could not find container \"09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d\": container with ID starting with 09e6193455c0fac5e7b12db852893c5b27ff3632a73ea48105d9bed958fb5c2d not found: ID does not exist"
Sep 17 18:47:00.020664 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:00.020576    2580 scope.go:117] "RemoveContainer" containerID="2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd"
Sep 17 18:47:00.021561 ip-10-0-139-39 kubenswrapper[2580]: E0917 18:47:00.021530    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd\": container with ID starting with 2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd not found: ID does not exist" containerID="2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd"
Sep 17 18:47:00.021764 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:00.021737    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd"} err="failed to get container status \"2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd\": rpc error: code = NotFound desc = could not find container \"2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd\": container with ID starting with 2c4e0081417c407b3e249a6dbfa419d1e0b4e6a145e1c4d28b5cc9d889f3d0dd not found: ID does not exist"
Sep 17 18:47:00.055021 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:00.054980    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" path="/var/lib/kubelet/pods/205c2c8d-202a-41c8-8e1a-fb99f184298b/volumes"
Sep 17 18:47:00.319365 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:00.319225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:00.456283 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:00.456248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6dnj_40439aa5-ea91-4890-8db7-7504f4047dff/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:02.624146 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:02.624039    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-rk659_b6cef6a2-dfb4-4399-9b2f-0ea208409579/manager/0.log"
Sep 17 18:47:02.681447 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:02.681413    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wzhjg_16466b21-0f3d-49c6-abef-4d4c9c15a7ee/limitador/0.log"
Sep 17 18:47:02.722216 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:02.722175    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-k7bhn_c6db2741-11b5-4284-886b-a6829b728d09/manager/0.log"
Sep 17 18:47:04.495006 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:04.494959    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-xqcxc_a0c85c0e-7126-471d-bd1f-292693933089/cluster-monitoring-operator/0.log"
Sep 17 18:47:04.754391 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:04.754286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q868m_a101289d-9b18-456f-84d7-86a34dab5d74/node-exporter/0.log"
Sep 17 18:47:04.777314 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:04.777272    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q868m_a101289d-9b18-456f-84d7-86a34dab5d74/kube-rbac-proxy/0.log"
Sep 17 18:47:04.803272 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:04.803240    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q868m_a101289d-9b18-456f-84d7-86a34dab5d74/init-textfile/0.log"
Sep 17 18:47:05.223184 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:05.223068    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-gv9n6_21bd49d9-5444-4a12-9481-885bfad0e3e7/prometheus-operator-admission-webhook/0.log"
Sep 17 18:47:07.156337 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:07.156302    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-fb6ww_44df5c8b-3917-43fa-b640-104adae7fec0/networking-console-plugin/0.log"
Sep 17 18:47:07.700214 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:07.700181    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/1.log"
Sep 17 18:47:07.707900 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:07.707859    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jnfbk_feae8d3b-b661-4b65-8713-fbf8404a86ed/console-operator/2.log"
Sep 17 18:47:08.223299 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.223262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-86lrx_cca89106-6d22-480b-a8b9-b76c470fd166/download-server/0.log"
Sep 17 18:47:08.554008 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.553912    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"]
Sep 17 18:47:08.554857 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.554824    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerName="gather"
Sep 17 18:47:08.555030 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.555016    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerName="gather"
Sep 17 18:47:08.555190 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.555175    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerName="copy"
Sep 17 18:47:08.555295 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.555283    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerName="copy"
Sep 17 18:47:08.555512 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.555496    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerName="gather"
Sep 17 18:47:08.555620 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.555607    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="205c2c8d-202a-41c8-8e1a-fb99f184298b" containerName="copy"
Sep 17 18:47:08.563219 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.563188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.567761 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.567723    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"]
Sep 17 18:47:08.656877 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.656825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-sys\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.657242 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.657213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-proc\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.657478 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.657457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzcv\" (UniqueName: \"kubernetes.io/projected/de88a04e-93e6-4bea-a76c-4729133679f3-kube-api-access-kfzcv\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.657647 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.657619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-lib-modules\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.657883 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.657858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-podres\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759055 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.758995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzcv\" (UniqueName: \"kubernetes.io/projected/de88a04e-93e6-4bea-a76c-4729133679f3-kube-api-access-kfzcv\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759055 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-lib-modules\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-podres\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-sys\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759355 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-proc\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759537 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-proc\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759797 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-sys\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.759992 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-podres\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.760135 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.759878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de88a04e-93e6-4bea-a76c-4729133679f3-lib-modules\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.767819 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.767788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzcv\" (UniqueName: \"kubernetes.io/projected/de88a04e-93e6-4bea-a76c-4729133679f3-kube-api-access-kfzcv\") pod \"perf-node-gather-daemonset-bgld7\" (UID: \"de88a04e-93e6-4bea-a76c-4729133679f3\") " pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:08.847949 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.847863    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-mfj5q_36932b3a-b41e-4d15-b034-170f2ea4c6a8/volume-data-source-validator/0.log"
Sep 17 18:47:08.877616 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:08.877574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:09.190332 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:09.188631    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"]
Sep 17 18:47:10.017722 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.017690    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-blzqz_24a97cf2-225e-4096-9b32-05fe7c4f9e75/dns/0.log"
Sep 17 18:47:10.026059 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.026021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7" event={"ID":"de88a04e-93e6-4bea-a76c-4729133679f3","Type":"ContainerStarted","Data":"ff6d09c48fdac5863abafdde5aad3dc1aa4077d7922870e587fe6238b13b1603"}
Sep 17 18:47:10.026244 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.026098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7" event={"ID":"de88a04e-93e6-4bea-a76c-4729133679f3","Type":"ContainerStarted","Data":"241ef69dcf587f8034499f654d304bc27affe55aee3e9205d3affd309a9b80c8"}
Sep 17 18:47:10.026244 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.026151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:10.037240 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.037196    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-blzqz_24a97cf2-225e-4096-9b32-05fe7c4f9e75/kube-rbac-proxy/0.log"
Sep 17 18:47:10.043480 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.043428    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7" podStartSLOduration=2.043411069 podStartE2EDuration="2.043411069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:10.042658354 +0000 UTC m=+4006.382051951" watchObservedRunningTime="2026-09-17 18:47:10.043411069 +0000 UTC m=+4006.382804663"
Sep 17 18:47:10.103593 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.103558    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f2lj4_23fef4bc-4131-4d25-b641-686631df7b5e/dns-node-resolver/0.log"
Sep 17 18:47:10.690718 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:10.690666    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hxlp_3f49d3fa-4582-4508-8858-4596395b5f57/node-ca/0.log"
Sep 17 18:47:11.811656 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:11.811624    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79d7b547b8-gfldz_63f2f4e6-aba2-4b69-8ef9-f2b68013d13e/router/0.log"
Sep 17 18:47:12.426169 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:12.426129    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j58hh_bd00e048-2992-4249-9ffd-d4d56bb6b446/serve-healthcheck-canary/0.log"
Sep 17 18:47:12.950528 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:12.950444    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-gczxt_f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6/insights-operator/0.log"
Sep 17 18:47:12.952277 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:12.952243    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-gczxt_f1331c2d-f21b-4bfa-a2a2-af5217bb1ef6/insights-operator/1.log"
Sep 17 18:47:12.981073 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:12.981042    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7dtsh_a2f72097-38a0-4d9c-bace-cffca394ce07/kube-rbac-proxy/0.log"
Sep 17 18:47:13.032323 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:13.032268    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7dtsh_a2f72097-38a0-4d9c-bace-cffca394ce07/exporter/0.log"
Sep 17 18:47:13.090542 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:13.090500    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7dtsh_a2f72097-38a0-4d9c-bace-cffca394ce07/extractor/0.log"
Sep 17 18:47:16.046684 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:16.046652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rx6xt/perf-node-gather-daemonset-bgld7"
Sep 17 18:47:16.898522 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:16.898487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5c5cdf4877-7zmcc_31a1faa1-7a6b-43d3-8d10-5b571127690c/manager/0.log"
Sep 17 18:47:17.366162 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:17.365559    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-7c5846b585-ckxfg_98cfa362-bc49-4abd-9574-8649756011c7/manager/0.log"
Sep 17 18:47:17.389703 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:17.389670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-l559l_2b610af7-cb50-4b55-9e6f-faf5e01adc31/server/0.log"
Sep 17 18:47:18.502195 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:18.502152    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-8b9rk_6f96617e-b442-43c2-aed7-b1cb5482b061/manager/0.log"
Sep 17 18:47:18.550061 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:18.549988    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ms4rf_f2292953-2b2b-4463-bba2-dff327fe7cd9/s3-init/0.log"
Sep 17 18:47:18.570016 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:18.569943    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ms4rf_f2292953-2b2b-4463-bba2-dff327fe7cd9/download-hf-model/0.log"
Sep 17 18:47:18.609261 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:18.609226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-sgccl_93140f91-8283-48a7-80b6-1399328a3123/seaweedfs/0.log"
Sep 17 18:47:18.681958 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:18.681927    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79d7b547b8-gfldz_63f2f4e6-aba2-4b69-8ef9-f2b68013d13e/router/0.log"
Sep 17 18:47:19.957068 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:19.957027    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79d7b547b8-gfldz_63f2f4e6-aba2-4b69-8ef9-f2b68013d13e/router/0.log"
Sep 17 18:47:24.201849 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:24.201809    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-dmz2q_f6481e0f-0af5-42c7-86ac-0c268a08bb19/kube-storage-version-migrator-operator/1.log"
Sep 17 18:47:24.204056 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:24.204026    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-dmz2q_f6481e0f-0af5-42c7-86ac-0c268a08bb19/kube-storage-version-migrator-operator/0.log"
Sep 17 18:47:25.446384 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.446355    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tz442_4a333d12-515a-4448-a372-27444bf5cb3f/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:47:25.468377 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.468337    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tz442_4a333d12-515a-4448-a372-27444bf5cb3f/egress-router-binary-copy/0.log"
Sep 17 18:47:25.490570 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.490533    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tz442_4a333d12-515a-4448-a372-27444bf5cb3f/cni-plugins/0.log"
Sep 17 18:47:25.536333 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.536294    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tz442_4a333d12-515a-4448-a372-27444bf5cb3f/bond-cni-plugin/0.log"
Sep 17 18:47:25.553657 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.553621    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tz442_4a333d12-515a-4448-a372-27444bf5cb3f/routeoverride-cni/0.log"
Sep 17 18:47:25.574616 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.574585    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tz442_4a333d12-515a-4448-a372-27444bf5cb3f/whereabouts-cni-bincopy/0.log"
Sep 17 18:47:25.595073 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.595045    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tz442_4a333d12-515a-4448-a372-27444bf5cb3f/whereabouts-cni/0.log"
Sep 17 18:47:25.889565 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.889480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tx7b2_ddfa57a6-109d-430c-b75d-cccf0b616d95/kube-multus/0.log"
Sep 17 18:47:25.960489 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.960455    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d67rh_78c77eba-b3f9-4472-8dad-5b68b500e7dd/network-metrics-daemon/0.log"
Sep 17 18:47:25.974927 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:25.974889    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d67rh_78c77eba-b3f9-4472-8dad-5b68b500e7dd/kube-rbac-proxy/0.log"
Sep 17 18:47:26.861474 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:26.861429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/ovn-controller/0.log"
Sep 17 18:47:26.879494 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:26.879460    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/ovn-acl-logging/0.log"
Sep 17 18:47:26.909366 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:26.909314    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/kube-rbac-proxy-node/0.log"
Sep 17 18:47:26.930342 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:26.930306    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:47:26.948920 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:26.948872    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/northd/0.log"
Sep 17 18:47:26.969072 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:26.969035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/nbdb/0.log"
Sep 17 18:47:26.989617 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:26.989585    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/sbdb/0.log"
Sep 17 18:47:27.098633 ip-10-0-139-39 kubenswrapper[2580]: I0917 18:47:27.098583    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4mftf_79df6c0a-271d-41dd-aab9-02fbedb94de1/ovnkube-controller/0.log"