Jul 28 17:14:47.322190 ip-10-0-135-20 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:14:47.322201 ip-10-0-135-20 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:14:47.322208 ip-10-0-135-20 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:14:47.322533 ip-10-0-135-20 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:14:58.905342 ip-10-0-135-20 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:14:58.905358 ip-10-0-135-20 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e5434dd8438542449794c938a1e3408f --
Jul 28 17:16:53.199205 ip-10-0-135-20 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:16:53.569779 ip-10-0-135-20 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:53.569779 ip-10-0-135-20 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:16:53.569779 ip-10-0-135-20 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:53.569779 ip-10-0-135-20 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:16:53.569779 ip-10-0-135-20 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:53.572096 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.571965    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:16:53.575810 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575795    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575812    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575818    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575821    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575825    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575828    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575832    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575834    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575837    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575840    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575843    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575845    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575848    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575850    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575853    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575856    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:53.575851 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575859    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575863    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575868    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575871    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575874    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575878    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575880    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575883    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575886    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575889    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575892    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575895    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575897    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575900    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575903    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575906    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575921    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575925    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575929    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575933    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:53.576255 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575936    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575938    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575941    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575943    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575946    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575948    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575951    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575953    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575956    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575958    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575960    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575963    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575966    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575970    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575972    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575975    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575979    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575982    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575985    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575988    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:53.576738 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575990    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575993    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575995    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.575998    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576000    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576003    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576005    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576008    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576010    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576013    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576015    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576018    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576020    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576023    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576026    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576028    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576031    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576033    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576035    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576038    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:53.577306 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576040    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576042    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576045    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576048    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576051    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576053    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576057    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576060    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576063    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576066    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576068    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576071    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576073    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576076    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576078    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576081    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576083    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576087    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576090    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:53.577791 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576093    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576097    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576099    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576102    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576773    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576779    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576781    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576785    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576788    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576790    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576793    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576796    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576798    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576801    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576803    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576806    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576808    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576812    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576814    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576817    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:53.578274 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576820    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576824    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576828    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576831    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576834    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576838    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576841    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576844    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576847    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576850    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576853    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576856    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576858    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576861    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576863    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576866    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576869    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576871    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576874    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:53.578779 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576876    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576879    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576881    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576884    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576886    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576889    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576892    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576895    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576897    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576899    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576902    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576905    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576926    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576930    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576933    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576936    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576939    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576942    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576945    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576948    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:53.579266 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576951    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576953    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576956    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576958    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576961    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576963    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576966    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576968    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576971    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576973    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576976    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576978    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576981    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576983    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576987    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576990    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576992    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576995    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.576998    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:53.579750 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577000    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577002    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577005    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577008    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577010    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577013    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577015    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577018    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577023    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577026    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577028    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577031    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577035    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577037    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577040    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577043    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577046    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577048    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577051    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577053    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:53.580234 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577056    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577060    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577063    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577066    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577069    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577456    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577461    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577464    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577467    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577470    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577473    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577476    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577478    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577480    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577483    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577486    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577489    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577491    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577495    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577497    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:53.580712 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577502    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577505    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577507    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577510    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577512    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577515    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577518    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577520    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577523    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577525    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577528    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577532    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577535    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577538    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577541    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577543    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577546    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577548    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577551    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:53.581224 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577554    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577558    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577561    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577564    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577566    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577569    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577571    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577574    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577577    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577579    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577582    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577584    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577587    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577593    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577596    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577598    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577601    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577604    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577607    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577610    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:53.581717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577613    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577615    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577618    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577622    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577625    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577628    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577631    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577634    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577637    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577640    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577642    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577644    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577647    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577649    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577652    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577654    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577657    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577660    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:53.582223 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577662    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577668    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577671    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577674    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577676    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577679    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577682    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577685    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577688    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577690    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577693    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577696    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577699    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577701    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577704    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577707    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577709    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577712    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577714    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577717    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:53.582671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577720    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577722    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577725    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577727    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577730    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577732    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.577735    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577796    2585 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577802    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577815    2585 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577819    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577823    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577827    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577832    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577836    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577839    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577842    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577846    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577849    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577854    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577857    2585 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577860    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:16:53.583197 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577863    2585 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577866    2585 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577869    2585 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577873    2585 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577876    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577880    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577883    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577886    2585 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577890    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577893    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577896    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577899    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577902    2585 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577905    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577923    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577928    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577933    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577938    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577941    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577944    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577947    2585 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577951    2585 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577953    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577960    2585 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577963    2585 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:16:53.583743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577967    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577970    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577972    2585 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577976    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577980    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577983    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577987    2585 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577991    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577993    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577996    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.577999    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578002    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578005    2585 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578008    2585 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578012    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578015    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578018    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578021    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578024    2585 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578027    2585 flags.go:64] FLAG: --help="false"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578029    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-135-20.ec2.internal"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578032    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578035    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:16:53.584405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578038    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578042    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578045    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578048    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578050    2585 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578053    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578056    2585 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578060    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578063    2585 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578066    2585 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578069    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578072    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578075    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578079    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578082    2585 flags.go:64] FLAG: --lock-file=""
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578085    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578087    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578090    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578095    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578098    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578101    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578104    2585 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578106    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578110    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:16:53.585028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578113    2585 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578116    2585 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578120    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578123    2585 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578127    2585 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578129    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578132    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578135    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578138    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578141    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578144    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578147    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578154    2585 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578157    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578160    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578164    2585 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578167    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578172    2585 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578175    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578178    2585 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578181    2585 flags.go:64] FLAG: --port="10250"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578185    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578188    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0794fb568f3850120"
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578191    2585 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:16:53.585932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578194    2585 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578196    2585 flags.go:64] FLAG: --register-node="true"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578200    2585 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578204    2585 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578207    2585 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578210    2585 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578213    2585 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578216    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578220    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578223    2585 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578226    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578229    2585 flags.go:64] FLAG: --runonce="false"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578231    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578234    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578237    2585 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578240    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578243    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578246    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578249    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578252    2585 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578255    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578258    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578260    2585 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578263    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578267    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:16:53.586672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578270    2585 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578273    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578278    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578281    2585 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578286    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578290    2585 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578293    2585 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578296    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578299    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578302    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578304    2585 flags.go:64] FLAG: --v="2"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578308    2585 flags.go:64] FLAG: --version="false"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578312    2585 flags.go:64] FLAG: --vmodule=""
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578316    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578319    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578440    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578445    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578449    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578453    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578456    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578459    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578462    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:53.587332 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578465    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578468    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578471    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578474    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578477    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578479    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578482    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578485    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578487    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578490    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578493    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578496    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578498    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578501    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578505    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578508    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578510    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578513    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578515    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578518    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:53.587871 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578520    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578523    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578525    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578528    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578531    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578533    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578536    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578539    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578541    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578544    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578546    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578549    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578551    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578554    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578556    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578559    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578561    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578563    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578566    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578568    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:53.588382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578571    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578574    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578576    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578579    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578581    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578585    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578590    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578593    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578597    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578601    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578603    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578606    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578609    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578612    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578614    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578617    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578619    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578622    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578625    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:53.588900 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578627    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578630    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578633    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578635    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578638    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578640    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578643    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578645    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578648    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578650    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578653    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578655    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578658    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578660    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578663    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578665    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578668    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578670    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578673    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578677    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:53.589381 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578679    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578682    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578684    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578687    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578689    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578691    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578694    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578696    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578699    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578701    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578704    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578706    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.578709    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.578714    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.587249    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:16:53.589870 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.587265    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.587284    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.587290    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587346    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587350    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587354    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587359    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587363    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587367    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587370    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587372    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587375    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587378    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587380    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587383    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587386    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587389    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587392    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587395    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:53.590257 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587399    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587403    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587406    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587408    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587411    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587414    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587417    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587420    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587422    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587425    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587427    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587430    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587433    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587436    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587439    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587441    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587444    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587447    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587449    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587452    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:53.590720 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587455    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587457    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587460    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587463    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587465    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587468    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587470    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587473    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587475    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587478    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587481    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587483    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587486    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587488    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587491    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587493    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587496    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587498    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587501    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587503    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:53.591228 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587506    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587509    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587512    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587514    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587518    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587521    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587523    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587526    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587529    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587531    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587534    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587536    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587539    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587541    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587544    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587546    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587549    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587551    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587554    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587556    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:53.591775 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587559    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587561    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587564    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587567    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587569    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587572    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587574    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587577    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587579    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587583    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587586    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587589    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587592    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587595    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587598    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587601    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587604    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587607    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587610    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:53.592325 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587612    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587615    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587618    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587621    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.587626    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587735    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587740    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587743    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587746    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587750    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587752    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587756    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587759    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587762    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587764    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:53.592805 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587767    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587770    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587773    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587775    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587778    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587781    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587784    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587786    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587789    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587791    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587794    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587796    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587799    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587801    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587804    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587807    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587809    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587812    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587814    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587817    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:53.593195 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587819    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587822    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587825    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587827    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587830    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587832    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587835    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587838    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587840    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587843    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587845    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587848    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587850    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587853    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587855    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587859    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587862    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587866    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587870    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587873    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:53.593676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587876    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587879    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587882    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587885    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587888    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587891    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587893    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587896    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587899    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587901    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587904    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587906    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587925    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587929    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587931    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587934    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587936    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587939    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587941    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587944    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:53.594177 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587947    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587949    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587952    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587955    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587957    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587959    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587962    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587964    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587969    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587973    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587976    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587979    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587981    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587984    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587986    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587989    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587991    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587994    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587997    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.587999    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:53.594677 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588001    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588004    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588006    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588009    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588011    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588014    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588017    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588020    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:53.588024    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.588029    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.588127    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.591188    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.592426    2585 server.go:1019] "Starting client certificate rotation"
Jul 28 17:16:53.595200 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.592521    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:16:53.595551 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.592550    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:16:53.595551 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.594885    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:16:53.597877 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.597860    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:16:53.610934 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.610903    2585 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:16:53.613233 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.613219    2585 log.go:25] "Validated CRI v1 image API"
Jul 28 17:16:53.613969 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.613956    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:16:53.616148 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.616129    2585 fs.go:135] Filesystem UUIDs: map[5f976269-d25c-46c2-88f1-1056f7503a8a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 82d789e2-d3f6-4ebc-bc52-802a712b5cf9:/dev/nvme0n1p4]
Jul 28 17:16:53.616196 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.616148    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:16:53.621743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.621720    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:16:53.622168 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.622060    2585 manager.go:216] Machine: {Timestamp:2026-07-28 17:16:53.619999529 +0000 UTC m=+0.310231681 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096944 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dafd592a72245b46eeebf3b31ffce SystemUUID:ec2dafd5-92a7-2245-b46e-eebf3b31ffce BootID:e5434dd8-4385-4244-9794-c938a1e3408f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6a:c0:24:7c:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6a:c0:24:7c:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:32:34:07:73:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:16:53.622168 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.622167    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:16:53.622266 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.622249    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:16:53.624794 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.624774    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:16:53.624941 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.624797    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-20.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:16:53.625563 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.625553    2585 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:16:53.625598 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.625566    2585 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:16:53.625598 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.625573    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:16:53.626140 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.626106    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:16:53.627421 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.627410    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:16:53.629162 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.629150    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:16:53.629472 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.629452    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:16:53.629572 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.629546    2585 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:16:53.629681 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.629589    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:16:53.629681 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.629661    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:16:53.629681 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.629677    2585 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:16:53.629801 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.629690    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:16:53.630258 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.630241    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:16:53.630328 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.630270    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:16:53.633012 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.632988    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:16:53.635256 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.635242    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:16:53.635311 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.635271    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:16:53.636792 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636780    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636796    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636802    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636808    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636816    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636821    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636828    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636833    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:16:53.636839 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636840    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:16:53.637057 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636845    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:16:53.637057 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636853    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:16:53.637057 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.636858    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:16:53.637719 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.637709    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:16:53.637719 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.637718    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:16:53.639717 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.639692    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:16:53.639795 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.639731    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-20.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:16:53.639955 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.639932    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:16:53.640855 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.640842    2585 server.go:1268] "Started kubelet"
Jul 28 17:16:53.640926 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.640885    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:16:53.640994 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.640946    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:16:53.641046 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.641033    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:16:53.641654 ip-10-0-135-20 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:16:53.643010 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.642995    2585 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:16:53.643169 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.643153    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:16:53.646463 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.646430    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:16:53.646463 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.646455    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n2ql5"
Jul 28 17:16:53.651755 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.651739    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:16:53.651817 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.651742    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:16:53.651866 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.651823    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:53.652386 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.652360    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:16:53.652386 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.652362    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:16:53.652386 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.652388    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:16:53.652620 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.652446    2585 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:16:53.652620 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.652457    2585 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:16:53.652771 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.652758    2585 factory.go:55] Registering systemd factory
Jul 28 17:16:53.652843 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.652775    2585 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:16:53.653051 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.653014    2585 factory.go:153] Registering CRI-O factory
Jul 28 17:16:53.653127 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.653093    2585 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:16:53.653217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.653143    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:16:53.653217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.653166    2585 factory.go:103] Registering Raw factory
Jul 28 17:16:53.653217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.653182    2585 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:16:53.653642 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.653611    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n2ql5"
Jul 28 17:16:53.653943 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.653929    2585 manager.go:349] Starting recovery of all containers
Jul 28 17:16:53.656001 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.655972    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:16:53.657160 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.656238    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-20.ec2.internal.18c6839d3c200199  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-20.ec2.internal,UID:ip-10-0-135-20.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-20.ec2.internal,},FirstTimestamp:2026-07-28 17:16:53.640815001 +0000 UTC m=+0.331047152,LastTimestamp:2026-07-28 17:16:53.640815001 +0000 UTC m=+0.331047152,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-20.ec2.internal,}"
Jul 28 17:16:53.660841 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.660812    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:16:53.665042 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.664903    2585 manager.go:354] Recovery completed
Jul 28 17:16:53.669720 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.669705    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:53.671408 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.671392    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:53.671487 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.671427    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:53.671487 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.671442    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:53.672020 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.672006    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:16:53.672020 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.672019    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:16:53.672107 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.672033    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:16:53.672107 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.672077    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:53.672719 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.672703    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:53.672793 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.672727    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:53.672793 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.672737    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:53.673669 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.673654    2585 policy_none.go:49] "None policy: Start"
Jul 28 17:16:53.673715 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.673671    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:16:53.673715 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.673682    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:16:53.674700 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.674689    2585 policy_none.go:47] "Start"
Jul 28 17:16:53.708640 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.708482    2585 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:16:53.708735 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.708665    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:16:53.708735 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.708677    2585 server.go:87] "Starting device plugin registration server"
Jul 28 17:16:53.708870 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.708860    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:16:53.708931 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.708870    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:16:53.709157 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.709130    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:16:53.709254 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.709246    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:16:53.709317 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.709256    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:16:53.709627 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.709607    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:16:53.709717 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.709652    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:53.809121 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.809093    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:53.810101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.809898    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:53.810101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.809940    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:53.810101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.809951    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:53.810101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.809976    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-20.ec2.internal"
Jul 28 17:16:53.820319 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.820267    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-20.ec2.internal"
Jul 28 17:16:53.820319 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.820291    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-20.ec2.internal\": node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:53.826616 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.826589    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:16:53.827766 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.827746    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:16:53.827841 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.827780    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:16:53.827841 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.827797    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:16:53.827841 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.827833    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:16:53.829631 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.829610    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:53.836312 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.836295    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:53.927954 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.927897    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"]
Jul 28 17:16:53.928105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.927991    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:53.928962 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.928930    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:53.928962 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.928958    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:53.928962 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.928967    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:53.930353 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.930341    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:53.930495 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.930471    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:53.930555 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.930518    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:53.931082 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.931067    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:53.931180 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.931092    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:53.931180 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.931068    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:53.931180 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.931126    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:53.931180 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.931146    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:53.931180 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.931101    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:53.932886 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.932871    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:53.932971 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.932896    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:53.933562 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.933547    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:53.933641 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.933576    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:53.933641 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:53.933588    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:53.936454 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.936436    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:53.968439 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.968419    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-20.ec2.internal\" not found" node="ip-10-0-135-20.ec2.internal"
Jul 28 17:16:53.972705 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:53.972690    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-20.ec2.internal\" not found" node="ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.383139 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:55.383123    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:55.399388 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.399366    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/789ac91acd8f458997b00ba308154756-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal\" (UID: \"789ac91acd8f458997b00ba308154756\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.399474 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.399404    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/789ac91acd8f458997b00ba308154756-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal\" (UID: \"789ac91acd8f458997b00ba308154756\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.399474 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.399425    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dd2e9a077ed553dd3893a28c3e87410-config\") pod \"kube-apiserver-proxy-ip-10-0-135-20.ec2.internal\" (UID: \"0dd2e9a077ed553dd3893a28c3e87410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.483768 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:55.483700    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:55.499999 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.499976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/789ac91acd8f458997b00ba308154756-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal\" (UID: \"789ac91acd8f458997b00ba308154756\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.500092 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.500029    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/789ac91acd8f458997b00ba308154756-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal\" (UID: \"789ac91acd8f458997b00ba308154756\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.500092 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.500074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/789ac91acd8f458997b00ba308154756-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal\" (UID: \"789ac91acd8f458997b00ba308154756\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.500173 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.500106    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dd2e9a077ed553dd3893a28c3e87410-config\") pod \"kube-apiserver-proxy-ip-10-0-135-20.ec2.internal\" (UID: \"0dd2e9a077ed553dd3893a28c3e87410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.500173 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.500134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dd2e9a077ed553dd3893a28c3e87410-config\") pod \"kube-apiserver-proxy-ip-10-0-135-20.ec2.internal\" (UID: \"0dd2e9a077ed553dd3893a28c3e87410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.500173 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.500152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/789ac91acd8f458997b00ba308154756-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal\" (UID: \"789ac91acd8f458997b00ba308154756\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.584164 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:55.584130    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:55.618536 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.618514    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.623661 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.623642    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.684282 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:55.684253    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:55.784720 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:55.784656    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-20.ec2.internal\" not found"
Jul 28 17:16:55.817287 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.817265    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:55.899061 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.899035    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.907851 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.907830    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:16:55.909649 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.909636    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal"
Jul 28 17:16:55.911000 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.910985    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:55.919456 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.919433    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:16:55.938722 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.938702    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:16:55.938857 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.938819    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:16:55.938955 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.938859    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:16:55.938955 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.938861    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:16:55.978042 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.978028    2585 apiserver.go:52] "Watching apiserver"
Jul 28 17:16:55.995388 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.995370    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:16:55.995741 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.995720    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal","openshift-multus/multus-additional-cni-plugins-8xzdj","openshift-multus/multus-f6vs6","openshift-network-diagnostics/network-check-target-4gsqc","openshift-ovn-kubernetes/ovnkube-node-kxrll","kube-system/konnectivity-agent-fz847","openshift-cluster-node-tuning-operator/tuned-xxpbg","openshift-dns/node-resolver-8gwcq","openshift-image-registry/node-ca-7nqzl","openshift-multus/network-metrics-daemon-ng94v","openshift-network-operator/iptables-alerter-gqqgn","kube-system/global-pull-secret-syncer-27fcp","kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal"]
Jul 28 17:16:55.997157 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.997142    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:55.999835 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.998846    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:16:55.999835 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.998902    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:55.999835 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.999823    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.000031 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:55.999945    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:16:56.000158 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.000140    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:16:56.000222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.000164    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:16:56.000302 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.000284    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8p6k\""
Jul 28 17:16:56.001352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001301    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-system-cni-dir\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.001352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001340    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-os-release\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.001531 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001364    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:16:56.001531 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001363    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:16:56.001531 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001366    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx7q\" (UniqueName: \"kubernetes.io/projected/10748634-0be3-4fd0-8789-da7427282953-kube-api-access-gxx7q\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.001531 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-cnibin\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.001739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001530    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.001739 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.001496    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:16:56.001739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001579    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.001739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001597    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:16:56.001739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001623    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-cni-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.001739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-socket-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.001739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001723    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-sys-fs\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.001977 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001779    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-os-release\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.001977 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.001810    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:16:56.002166 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-socket-dir-parent\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002220 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002179    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-k8s-cni-cncf-io\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002220 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002196    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-cni-bin\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002220 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002212    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-etc-selinux\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.002361 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002231    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.002361 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002264    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-registration-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.002361 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002310    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-system-cni-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002361 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002348    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-cni-binary-copy\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002368    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002402    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002399    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-multus-certs\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002435    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-netns\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002450    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-hostroot\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002465    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-etc-kubernetes\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002479    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g55cr\""
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002487    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002500    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pch5h\""
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-kubelet\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002538 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002533    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-daemon-config\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002571    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-device-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002592    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002594    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84n6h\" (UniqueName: \"kubernetes.io/projected/91aee170-ad83-440e-8544-ccc56a8f216a-kube-api-access-84n6h\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002617    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-cnibin\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002638    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-cni-multus\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002660    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-conf-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002682    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk7wc\" (UniqueName: \"kubernetes.io/projected/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-kube-api-access-qk7wc\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.002939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.002708    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.003485 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.003470    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.004565 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.004539    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.005799 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.005780    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.005908 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.005893    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r6lkq\""
Jul 28 17:16:56.006179 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006144    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:16:56.006257 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006189    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:16:56.006257 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006200    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:16:56.006257 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006218    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:16:56.006257 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006190    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:16:56.006428 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006201    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:16:56.006428 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006353    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:11:54 +0000 UTC" deadline="2028-01-23 02:10:08.66146437 +0000 UTC"
Jul 28 17:16:56.006428 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.006385    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13040h53m12.655082682s"
Jul 28 17:16:56.007060 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.007045    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.007133 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.007076    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:16:56.007133 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.007084    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z9r66\""
Jul 28 17:16:56.007233 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.007205    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:16:56.008254 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.008228    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:16:56.008254 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.008247    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:16:56.008465 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.008448    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mtnmv\""
Jul 28 17:16:56.008465 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.008461    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.009289 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.009274    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ntdq4\""
Jul 28 17:16:56.009412 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.009395    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:16:56.009515 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.009500    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:16:56.009926 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.009910    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:56.009993 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.009957    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:16:56.010849 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.010835    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:16:56.011036 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.011023    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:16:56.011262 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.011248    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rspxk\""
Jul 28 17:16:56.011479 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.011466    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:16:56.011697 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.011682    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.012104 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.012088    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:16:56.012878 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.012858    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.012954 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.012934    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:16:56.013877 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.013862    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:16:56.013978 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.013962    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:16:56.014116 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.014096    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58r5s\""
Jul 28 17:16:56.014217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.014139    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:16:56.038086 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.038038    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m8rbn"
Jul 28 17:16:56.048925 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.048910    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m8rbn"
Jul 28 17:16:56.099796 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.099777    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:16:56.103191 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-cni-multus\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103291 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-conf-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103291 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103220    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk7wc\" (UniqueName: \"kubernetes.io/projected/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-kube-api-access-qk7wc\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103291 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103255    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-run-netns\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.103291 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103279    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysctl-d\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103285    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-conf-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103288    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-cni-multus\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103329    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-os-release\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103371    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-run\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103383    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-os-release\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103399    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07aa6d61-639a-4114-9f14-21ef6bfe55fd-tmp-dir\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7vf\" (UniqueName: \"kubernetes.io/projected/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-kube-api-access-px7vf\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.103441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-cnibin\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103459    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103483    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103506    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-cni-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103578    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-modprobe-d\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103588    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-cnibin\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-socket-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-sys-fs\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103636    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103661    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-socket-dir-parent\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103688    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-socket-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103689    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9bt\" (UniqueName: \"kubernetes.io/projected/3a35d96c-75c0-4979-9470-4a4fc570ed7d-kube-api-access-qf9bt\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103701    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-sys-fs\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103716    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-lib-modules\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-cni-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103761    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103787    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovnkube-script-lib\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.103821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-socket-dir-parent\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103804    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-host-slash\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-registration-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103854    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-system-cni-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103874    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-var-lib-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103911    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103919    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-registration-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103942    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-system-cni-dir\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103970    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07aa6d61-639a-4114-9f14-21ef6bfe55fd-hosts-file\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.103994    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-hostroot\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104015    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-systemd\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104031    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-kubelet\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104032    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-hostroot\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104052    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f4bfba2-733c-453a-a03d-02d4994797d4-dbus\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104056    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-kubelet\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104077    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-node-log\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.104483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-var-lib-kubelet\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104133    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5js\" (UniqueName: \"kubernetes.io/projected/4008895f-2b03-4540-96d8-773fea27114f-kube-api-access-wp5js\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104145    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a5be7c1-fe99-43a9-acd3-912ea1a5c61e-agent-certs\") pod \"konnectivity-agent-fz847\" (UID: \"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e\") " pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104168    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-cnibin\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104190    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovn-node-metrics-cert\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-kubernetes\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysctl-conf\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104241    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4008895f-2b03-4540-96d8-773fea27114f-tmp\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104257    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104264    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-cnibin\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104272    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-system-cni-dir\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104302    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-system-cni-dir\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104325    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-ovn\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104332    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104354    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovnkube-config\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104378    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysconfig\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg62f\" (UniqueName: \"kubernetes.io/projected/0a3ff851-6c38-46db-b020-57ae09a57aa3-kube-api-access-wg62f\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104431    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx7q\" (UniqueName: \"kubernetes.io/projected/10748634-0be3-4fd0-8789-da7427282953-kube-api-access-gxx7q\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104454    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-systemd\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104502    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-sys\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104526    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3e7698f-f10c-4566-9052-349b0610b252-serviceca\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104579    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-os-release\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104604    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-k8s-cni-cncf-io\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104627    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-cni-bin\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-cni-bin\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104672    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-var-lib-cni-bin\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104695    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-host\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-k8s-cni-cncf-io\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104676    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91aee170-ad83-440e-8544-ccc56a8f216a-os-release\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104712    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4008895f-2b03-4540-96d8-773fea27114f-etc-tuned\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104730    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-etc-selinux\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104750    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-slash\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.105959 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104765    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-cni-netd\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104780    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-cni-binary-copy\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104795    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-multus-certs\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104810    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f4bfba2-733c-453a-a03d-02d4994797d4-kubelet-config\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104837    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-etc-selinux\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104890    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104897    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-multus-certs\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104918    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3e7698f-f10c-4566-9052-349b0610b252-host\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104935    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8qt\" (UniqueName: \"kubernetes.io/projected/c3e7698f-f10c-4566-9052-349b0610b252-kube-api-access-kq8qt\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104949    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a5be7c1-fe99-43a9-acd3-912ea1a5c61e-konnectivity-ca\") pod \"konnectivity-agent-fz847\" (UID: \"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e\") " pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104963    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-iptables-alerter-script\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.104984    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-netns\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-etc-kubernetes\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105028    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-etc-kubernetes\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-host-run-netns\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105057    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105077    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-kubelet\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.106535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105103    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-etc-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105120    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-env-overrides\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105134    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qqz\" (UniqueName: \"kubernetes.io/projected/07aa6d61-639a-4114-9f14-21ef6bfe55fd-kube-api-access-r2qqz\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105167    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-daemon-config\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105182    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-systemd-units\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105203    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-log-socket\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105216    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-device-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105245    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-cni-binary-copy\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105270    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10748634-0be3-4fd0-8789-da7427282953-device-dir\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105348    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84n6h\" (UniqueName: \"kubernetes.io/projected/91aee170-ad83-440e-8544-ccc56a8f216a-kube-api-access-84n6h\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105680    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91aee170-ad83-440e-8544-ccc56a8f216a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.107105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.105708    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-multus-daemon-config\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.110003 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.109972    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:16:56.112962 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.112942    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk7wc\" (UniqueName: \"kubernetes.io/projected/1b95f9c4-e82e-4530-bfb1-740d8c4e8c22-kube-api-access-qk7wc\") pod \"multus-f6vs6\" (UID: \"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22\") " pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.115808 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.115768    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx7q\" (UniqueName: \"kubernetes.io/projected/10748634-0be3-4fd0-8789-da7427282953-kube-api-access-gxx7q\") pod \"aws-ebs-csi-driver-node-9wttr\" (UID: \"10748634-0be3-4fd0-8789-da7427282953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.116734 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.116705    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84n6h\" (UniqueName: \"kubernetes.io/projected/91aee170-ad83-440e-8544-ccc56a8f216a-kube-api-access-84n6h\") pod \"multus-additional-cni-plugins-8xzdj\" (UID: \"91aee170-ad83-440e-8544-ccc56a8f216a\") " pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.205785 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205760    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:16:56.205917 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205791    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-kubelet\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.205917 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-etc-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.205917 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205831    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-env-overrides\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.205917 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205846    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qqz\" (UniqueName: \"kubernetes.io/projected/07aa6d61-639a-4114-9f14-21ef6bfe55fd-kube-api-access-r2qqz\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.205917 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-etc-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.205917 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-kubelet\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206167 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.205985    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-systemd-units\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206167 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206018    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-log-socket\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206167 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206063    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206167 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206094    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-run-netns\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206167 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-log-socket\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206167 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206062    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-systemd-units\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206216    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysctl-d\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206241    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206243    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-run-netns\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206280    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-run\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206340    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-run\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysctl-d\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.206370    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206377    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07aa6d61-639a-4114-9f14-21ef6bfe55fd-tmp-dir\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206403    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7vf\" (UniqueName: \"kubernetes.io/projected/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-kube-api-access-px7vf\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.206432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-env-overrides\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.206455    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:56.706424972 +0000 UTC m=+2.050017258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-modprobe-d\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206488    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9bt\" (UniqueName: \"kubernetes.io/projected/3a35d96c-75c0-4979-9470-4a4fc570ed7d-kube-api-access-qf9bt\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206505    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-lib-modules\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206528    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovnkube-script-lib\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206565    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-host-slash\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206599    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-modprobe-d\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206609    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-host-slash\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206645    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-lib-modules\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-var-lib-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206698    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07aa6d61-639a-4114-9f14-21ef6bfe55fd-tmp-dir\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206701    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206714    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-var-lib-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206736    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07aa6d61-639a-4114-9f14-21ef6bfe55fd-hosts-file\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206741    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.206940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-systemd\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206809    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f4bfba2-733c-453a-a03d-02d4994797d4-dbus\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07aa6d61-639a-4114-9f14-21ef6bfe55fd-hosts-file\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-node-log\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206872    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f4bfba2-733c-453a-a03d-02d4994797d4-dbus\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206883    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-var-lib-kubelet\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206837    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-systemd\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.206907    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-var-lib-kubelet\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206907    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5js\" (UniqueName: \"kubernetes.io/projected/4008895f-2b03-4540-96d8-773fea27114f-kube-api-access-wp5js\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.206908    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-node-log\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.206962    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret podName:8f4bfba2-733c-453a-a03d-02d4994797d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:56.706946741 +0000 UTC m=+2.050538997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret") pod "global-pull-secret-syncer-27fcp" (UID: "8f4bfba2-733c-453a-a03d-02d4994797d4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207002    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a5be7c1-fe99-43a9-acd3-912ea1a5c61e-agent-certs\") pod \"konnectivity-agent-fz847\" (UID: \"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e\") " pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207031    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovn-node-metrics-cert\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-kubernetes\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207079    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysctl-conf\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.207578 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207099    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-kubernetes\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207056    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovnkube-script-lib\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysctl-conf\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207102    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4008895f-2b03-4540-96d8-773fea27114f-tmp\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-ovn\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovnkube-config\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207666    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-ovn\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207724    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysconfig\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207763    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg62f\" (UniqueName: \"kubernetes.io/projected/0a3ff851-6c38-46db-b020-57ae09a57aa3-kube-api-access-wg62f\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-systemd\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207820    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-etc-sysconfig\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-sys\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207853    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3e7698f-f10c-4566-9052-349b0610b252-serviceca\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207882    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-systemd\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207888    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-cni-bin\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207917    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-host\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207947    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4008895f-2b03-4540-96d8-773fea27114f-etc-tuned\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207982    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-slash\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.208366 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-cni-netd\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208099    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovnkube-config\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.207949    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-sys\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208483    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3e7698f-f10c-4566-9052-349b0610b252-serviceca\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208508    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-slash\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208546    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-cni-bin\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208577    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4008895f-2b03-4540-96d8-773fea27114f-host\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208673    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-host-cni-netd\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f4bfba2-733c-453a-a03d-02d4994797d4-kubelet-config\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208775    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3e7698f-f10c-4566-9052-349b0610b252-host\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208809    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8qt\" (UniqueName: \"kubernetes.io/projected/c3e7698f-f10c-4566-9052-349b0610b252-kube-api-access-kq8qt\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208843    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a5be7c1-fe99-43a9-acd3-912ea1a5c61e-konnectivity-ca\") pod \"konnectivity-agent-fz847\" (UID: \"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e\") " pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.209259 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.208874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-iptables-alerter-script\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.209877 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.209430    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-iptables-alerter-script\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.209877 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.209515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f4bfba2-733c-453a-a03d-02d4994797d4-kubelet-config\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.209877 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.209583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a35d96c-75c0-4979-9470-4a4fc570ed7d-run-openvswitch\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.209877 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.209631    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3e7698f-f10c-4566-9052-349b0610b252-host\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.210245 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.210209    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4008895f-2b03-4540-96d8-773fea27114f-tmp\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.210340 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.210322    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a35d96c-75c0-4979-9470-4a4fc570ed7d-ovn-node-metrics-cert\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.210413 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.210348    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a5be7c1-fe99-43a9-acd3-912ea1a5c61e-konnectivity-ca\") pod \"konnectivity-agent-fz847\" (UID: \"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e\") " pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.212543 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.212522    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a5be7c1-fe99-43a9-acd3-912ea1a5c61e-agent-certs\") pod \"konnectivity-agent-fz847\" (UID: \"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e\") " pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.213205 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.213166    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4008895f-2b03-4540-96d8-773fea27114f-etc-tuned\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.216207 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.216186    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:56.216350 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.216211    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:56.216350 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.216226    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ktt2f for pod openshift-network-diagnostics/network-check-target-4gsqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:56.216350 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.216287    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f podName:1d9f58fc-5928-4213-8e45-380962645bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:56.716269349 +0000 UTC m=+2.059861614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ktt2f" (UniqueName: "kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f") pod "network-check-target-4gsqc" (UID: "1d9f58fc-5928-4213-8e45-380962645bd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:56.217928 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.217875    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789ac91acd8f458997b00ba308154756.slice/crio-ba26d574bc681f35f98019241590f6f62e3e243dee9e609f1cc0952e446b772c WatchSource:0}: Error finding container ba26d574bc681f35f98019241590f6f62e3e243dee9e609f1cc0952e446b772c: Status 404 returned error can't find the container with id ba26d574bc681f35f98019241590f6f62e3e243dee9e609f1cc0952e446b772c
Jul 28 17:16:56.218640 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.218539    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9bt\" (UniqueName: \"kubernetes.io/projected/3a35d96c-75c0-4979-9470-4a4fc570ed7d-kube-api-access-qf9bt\") pod \"ovnkube-node-kxrll\" (UID: \"3a35d96c-75c0-4979-9470-4a4fc570ed7d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.218758 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.218660    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd2e9a077ed553dd3893a28c3e87410.slice/crio-f1ffa65ab1141b995adb8e0518218f19170ae925049a00f36c476b6638ed4a75 WatchSource:0}: Error finding container f1ffa65ab1141b995adb8e0518218f19170ae925049a00f36c476b6638ed4a75: Status 404 returned error can't find the container with id f1ffa65ab1141b995adb8e0518218f19170ae925049a00f36c476b6638ed4a75
Jul 28 17:16:56.218948 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.218921    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8qt\" (UniqueName: \"kubernetes.io/projected/c3e7698f-f10c-4566-9052-349b0610b252-kube-api-access-kq8qt\") pod \"node-ca-7nqzl\" (UID: \"c3e7698f-f10c-4566-9052-349b0610b252\") " pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.219232 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.219210    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5js\" (UniqueName: \"kubernetes.io/projected/4008895f-2b03-4540-96d8-773fea27114f-kube-api-access-wp5js\") pod \"tuned-xxpbg\" (UID: \"4008895f-2b03-4540-96d8-773fea27114f\") " pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.219427 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.219405    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qqz\" (UniqueName: \"kubernetes.io/projected/07aa6d61-639a-4114-9f14-21ef6bfe55fd-kube-api-access-r2qqz\") pod \"node-resolver-8gwcq\" (UID: \"07aa6d61-639a-4114-9f14-21ef6bfe55fd\") " pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.219733 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.219711    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg62f\" (UniqueName: \"kubernetes.io/projected/0a3ff851-6c38-46db-b020-57ae09a57aa3-kube-api-access-wg62f\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:56.220074 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.220054    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7vf\" (UniqueName: \"kubernetes.io/projected/e0ab3143-f0ea-44c3-b554-b78e9d8849b2-kube-api-access-px7vf\") pod \"iptables-alerter-gqqgn\" (UID: \"e0ab3143-f0ea-44c3-b554-b78e9d8849b2\") " pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.224047 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.224023    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:16:56.316422 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.316347    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr"
Jul 28 17:16:56.321302 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.321282    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xzdj"
Jul 28 17:16:56.326809 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.326786    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10748634_0be3_4fd0_8789_da7427282953.slice/crio-baf51c0ec204e6d841ac3f82e6d5ad597ced6dc77d63b4fe815040472896f8da WatchSource:0}: Error finding container baf51c0ec204e6d841ac3f82e6d5ad597ced6dc77d63b4fe815040472896f8da: Status 404 returned error can't find the container with id baf51c0ec204e6d841ac3f82e6d5ad597ced6dc77d63b4fe815040472896f8da
Jul 28 17:16:56.330292 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.330215    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aee170_ad83_440e_8544_ccc56a8f216a.slice/crio-f1897e86e909546d488e99159bc0d1d426bf641c58a5044da8b241fb528699fe WatchSource:0}: Error finding container f1897e86e909546d488e99159bc0d1d426bf641c58a5044da8b241fb528699fe: Status 404 returned error can't find the container with id f1897e86e909546d488e99159bc0d1d426bf641c58a5044da8b241fb528699fe
Jul 28 17:16:56.340807 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.340788    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:56.350805 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.350783    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6vs6"
Jul 28 17:16:56.357524 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.357495    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b95f9c4_e82e_4530_bfb1_740d8c4e8c22.slice/crio-c016a024305e9c003353f89359afdd613d4e9baa76e0cb453852f3afd33ee253 WatchSource:0}: Error finding container c016a024305e9c003353f89359afdd613d4e9baa76e0cb453852f3afd33ee253: Status 404 returned error can't find the container with id c016a024305e9c003353f89359afdd613d4e9baa76e0cb453852f3afd33ee253
Jul 28 17:16:56.362405 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.362387    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:16:56.368343 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.368322    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a35d96c_75c0_4979_9470_4a4fc570ed7d.slice/crio-c969a9a0590a2fdbd2db5a3d8c786ea81f8bc81038732f00a860d6805af6840c WatchSource:0}: Error finding container c969a9a0590a2fdbd2db5a3d8c786ea81f8bc81038732f00a860d6805af6840c: Status 404 returned error can't find the container with id c969a9a0590a2fdbd2db5a3d8c786ea81f8bc81038732f00a860d6805af6840c
Jul 28 17:16:56.379942 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.379922    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:16:56.384652 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.384632    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xxpbg"
Jul 28 17:16:56.386204 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.386187    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gwcq"
Jul 28 17:16:56.386351 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.386332    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5be7c1_fe99_43a9_acd3_912ea1a5c61e.slice/crio-e83ea362f684e5a1b7b7fd1f3dd273217349740e8595f17b53de53261e9db473 WatchSource:0}: Error finding container e83ea362f684e5a1b7b7fd1f3dd273217349740e8595f17b53de53261e9db473: Status 404 returned error can't find the container with id e83ea362f684e5a1b7b7fd1f3dd273217349740e8595f17b53de53261e9db473
Jul 28 17:16:56.391495 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.391478    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4008895f_2b03_4540_96d8_773fea27114f.slice/crio-659dec8edf1877f8367e1f0b439ec06430101d38db298328275c2dbb5b96bb2f WatchSource:0}: Error finding container 659dec8edf1877f8367e1f0b439ec06430101d38db298328275c2dbb5b96bb2f: Status 404 returned error can't find the container with id 659dec8edf1877f8367e1f0b439ec06430101d38db298328275c2dbb5b96bb2f
Jul 28 17:16:56.394131 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.394117    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07aa6d61_639a_4114_9f14_21ef6bfe55fd.slice/crio-f22c61d30a4e6a7239852eb75d4beaf222a1424693b0c5d44852a42e473a5f97 WatchSource:0}: Error finding container f22c61d30a4e6a7239852eb75d4beaf222a1424693b0c5d44852a42e473a5f97: Status 404 returned error can't find the container with id f22c61d30a4e6a7239852eb75d4beaf222a1424693b0c5d44852a42e473a5f97
Jul 28 17:16:56.412225 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.412205    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7nqzl"
Jul 28 17:16:56.419184 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.419164    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e7698f_f10c_4566_9052_349b0610b252.slice/crio-4948a5d9ca2130d1d8f6fce2b880f1f176885cabd6231095432a6f91f5bc930f WatchSource:0}: Error finding container 4948a5d9ca2130d1d8f6fce2b880f1f176885cabd6231095432a6f91f5bc930f: Status 404 returned error can't find the container with id 4948a5d9ca2130d1d8f6fce2b880f1f176885cabd6231095432a6f91f5bc930f
Jul 28 17:16:56.424960 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.424945    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqqgn"
Jul 28 17:16:56.431718 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:16:56.431694    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ab3143_f0ea_44c3_b554_b78e9d8849b2.slice/crio-d0f310e815f377967f028742b5dfb4b537b14ec383bc4baf406bff460a446736 WatchSource:0}: Error finding container d0f310e815f377967f028742b5dfb4b537b14ec383bc4baf406bff460a446736: Status 404 returned error can't find the container with id d0f310e815f377967f028742b5dfb4b537b14ec383bc4baf406bff460a446736
Jul 28 17:16:56.713376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.712642    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:56.713376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.712698    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:56.713376 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.712824    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:56.713376 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.712879    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret podName:8f4bfba2-733c-453a-a03d-02d4994797d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:57.712860474 +0000 UTC m=+3.056452730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret") pod "global-pull-secret-syncer-27fcp" (UID: "8f4bfba2-733c-453a-a03d-02d4994797d4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:56.713376 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.713252    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:56.713376 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.713297    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:57.713282956 +0000 UTC m=+3.056875214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:56.780728 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.780509    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:56.813714 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:56.813250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:16:56.813714 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.813388    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:56.813714 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.813404    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:56.813714 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.813412    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ktt2f for pod openshift-network-diagnostics/network-check-target-4gsqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:56.813714 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:56.813453    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f podName:1d9f58fc-5928-4213-8e45-380962645bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:57.813442187 +0000 UTC m=+3.157034442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktt2f" (UniqueName: "kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f") pod "network-check-target-4gsqc" (UID: "1d9f58fc-5928-4213-8e45-380962645bd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:57.051067 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.050960    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:11:56 +0000 UTC" deadline="2028-01-14 09:14:34.783287817 +0000 UTC"
Jul 28 17:16:57.051067 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.051022    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12831h57m37.732269542s"
Jul 28 17:16:57.175664 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.175633    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:57.175842 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.175766    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:16:57.212510 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.212412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gwcq" event={"ID":"07aa6d61-639a-4114-9f14-21ef6bfe55fd","Type":"ContainerStarted","Data":"f22c61d30a4e6a7239852eb75d4beaf222a1424693b0c5d44852a42e473a5f97"}
Jul 28 17:16:57.235812 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.235776    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerStarted","Data":"f1897e86e909546d488e99159bc0d1d426bf641c58a5044da8b241fb528699fe"}
Jul 28 17:16:57.252809 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.252775    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr" event={"ID":"10748634-0be3-4fd0-8789-da7427282953","Type":"ContainerStarted","Data":"baf51c0ec204e6d841ac3f82e6d5ad597ced6dc77d63b4fe815040472896f8da"}
Jul 28 17:16:57.289219 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.289132    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal" event={"ID":"0dd2e9a077ed553dd3893a28c3e87410","Type":"ContainerStarted","Data":"f1ffa65ab1141b995adb8e0518218f19170ae925049a00f36c476b6638ed4a75"}
Jul 28 17:16:57.295968 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.295938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqqgn" event={"ID":"e0ab3143-f0ea-44c3-b554-b78e9d8849b2","Type":"ContainerStarted","Data":"d0f310e815f377967f028742b5dfb4b537b14ec383bc4baf406bff460a446736"}
Jul 28 17:16:57.300631 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.300584    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7nqzl" event={"ID":"c3e7698f-f10c-4566-9052-349b0610b252","Type":"ContainerStarted","Data":"4948a5d9ca2130d1d8f6fce2b880f1f176885cabd6231095432a6f91f5bc930f"}
Jul 28 17:16:57.328204 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.328062    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xxpbg" event={"ID":"4008895f-2b03-4540-96d8-773fea27114f","Type":"ContainerStarted","Data":"659dec8edf1877f8367e1f0b439ec06430101d38db298328275c2dbb5b96bb2f"}
Jul 28 17:16:57.341042 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.340985    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fz847" event={"ID":"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e","Type":"ContainerStarted","Data":"e83ea362f684e5a1b7b7fd1f3dd273217349740e8595f17b53de53261e9db473"}
Jul 28 17:16:57.349048 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.349012    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"c969a9a0590a2fdbd2db5a3d8c786ea81f8bc81038732f00a860d6805af6840c"}
Jul 28 17:16:57.373688 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.373650    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6vs6" event={"ID":"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22","Type":"ContainerStarted","Data":"c016a024305e9c003353f89359afdd613d4e9baa76e0cb453852f3afd33ee253"}
Jul 28 17:16:57.392703 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.392670    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal" event={"ID":"789ac91acd8f458997b00ba308154756","Type":"ContainerStarted","Data":"ba26d574bc681f35f98019241590f6f62e3e243dee9e609f1cc0952e446b772c"}
Jul 28 17:16:57.720542 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.720422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:57.720542 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.720484    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:57.720779 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.720633    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:57.720779 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.720694    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret podName:8f4bfba2-733c-453a-a03d-02d4994797d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:59.720676548 +0000 UTC m=+5.064268818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret") pod "global-pull-secret-syncer-27fcp" (UID: "8f4bfba2-733c-453a-a03d-02d4994797d4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:57.721685 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.721116    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:57.721685 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.721169    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:59.72115286 +0000 UTC m=+5.064745129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:57.821698 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:57.821536    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:16:57.821900 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.821760    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:57.821900 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.821780    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:57.821900 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.821792    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ktt2f for pod openshift-network-diagnostics/network-check-target-4gsqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:57.821900 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:57.821876    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f podName:1d9f58fc-5928-4213-8e45-380962645bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:59.821830887 +0000 UTC m=+5.165423149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktt2f" (UniqueName: "kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f") pod "network-check-target-4gsqc" (UID: "1d9f58fc-5928-4213-8e45-380962645bd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:58.052131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:58.052012    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:11:56 +0000 UTC" deadline="2028-01-26 16:05:50.845270985 +0000 UTC"
Jul 28 17:16:58.052131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:58.052073    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13126h48m52.793202113s"
Jul 28 17:16:58.083695 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:58.083665    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:58.174802 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:58.174771    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:58.174981 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:58.174910    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:16:58.175317 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:58.175301    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:16:58.175406 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:58.175388    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:16:58.178164 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:58.178144    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:59.181705 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:59.176775    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:59.181705 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.176915    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:16:59.742089 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:59.741351    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:16:59.742089 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:59.741412    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:16:59.742089 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.741530    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:59.742089 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.741608    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret podName:8f4bfba2-733c-453a-a03d-02d4994797d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:03.741589907 +0000 UTC m=+9.085182167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret") pod "global-pull-secret-syncer-27fcp" (UID: "8f4bfba2-733c-453a-a03d-02d4994797d4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:59.742089 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.741988    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:59.742089 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.742056    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:03.74203999 +0000 UTC m=+9.085632251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:59.843339 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:16:59.842657    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:16:59.843339 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.842889    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:59.843339 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.842911    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:59.843339 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.842924    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ktt2f for pod openshift-network-diagnostics/network-check-target-4gsqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:59.843339 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:16:59.842979    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f podName:1d9f58fc-5928-4213-8e45-380962645bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:03.842961931 +0000 UTC m=+9.186554197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktt2f" (UniqueName: "kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f") pod "network-check-target-4gsqc" (UID: "1d9f58fc-5928-4213-8e45-380962645bd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:00.174941 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:00.174837    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:00.175109 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:00.174991    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:00.175109 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:00.174845    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:00.175219 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:00.175125    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:01.175678 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:01.175487    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:01.175678 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:01.175640    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:02.175255 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:02.175221    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:02.175452 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:02.175342    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:02.175452 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:02.175221    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:02.175588 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:02.175451    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:03.178198 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:03.178169    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:03.178605 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.178294    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:03.774087 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:03.774041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:03.774269 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:03.774116    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:03.774330 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.774293    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:03.774386 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.774344    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:03.774386 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.774365    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret podName:8f4bfba2-733c-453a-a03d-02d4994797d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:11.77434471 +0000 UTC m=+17.117936977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret") pod "global-pull-secret-syncer-27fcp" (UID: "8f4bfba2-733c-453a-a03d-02d4994797d4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:03.774495 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.774408    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:11.774389083 +0000 UTC m=+17.117981338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:03.875004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:03.874914    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:03.875186 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.875078    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:03.875186 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.875106    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:03.875186 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.875120    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ktt2f for pod openshift-network-diagnostics/network-check-target-4gsqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:03.875186 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:03.875186    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f podName:1d9f58fc-5928-4213-8e45-380962645bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:11.87516779 +0000 UTC m=+17.218760048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktt2f" (UniqueName: "kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f") pod "network-check-target-4gsqc" (UID: "1d9f58fc-5928-4213-8e45-380962645bd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:04.175598 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:04.175500    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:04.175598 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:04.175545    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:04.175831 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:04.175673    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:04.176006 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:04.175957    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:05.176613 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:05.176569    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:05.177082 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:05.176687    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:06.175264 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:06.175233    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:06.175420 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:06.175348    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:06.175420 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:06.175408    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:06.175544 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:06.175524    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:07.174733 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:07.174701    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:07.175215 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:07.174823    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:08.175032 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:08.175003    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:08.175469 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:08.175004    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:08.175469 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:08.175119    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:08.175469 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:08.175224    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:09.175483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:09.175435    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:09.175943 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:09.175579    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:10.175320 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:10.175285    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:10.175486 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:10.175301    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:10.175486 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:10.175435    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:10.175486 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:10.175461    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:11.174915 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:11.174875    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:11.175105 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.175028    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:11.836124 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:11.836098    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:11.836465 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:11.836136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:11.836465 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.836229    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:11.836465 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.836260    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:11.836465 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.836269    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret podName:8f4bfba2-733c-453a-a03d-02d4994797d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:27.836258068 +0000 UTC m=+33.179850327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret") pod "global-pull-secret-syncer-27fcp" (UID: "8f4bfba2-733c-453a-a03d-02d4994797d4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:11.836465 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.836328    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:27.836309782 +0000 UTC m=+33.179902048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:11.937265 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:11.937232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:11.937424 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.937366    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:11.937424 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.937384    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:11.937424 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.937395    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ktt2f for pod openshift-network-diagnostics/network-check-target-4gsqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:11.937602 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:11.937452    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f podName:1d9f58fc-5928-4213-8e45-380962645bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:27.937434026 +0000 UTC m=+33.281026285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktt2f" (UniqueName: "kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f") pod "network-check-target-4gsqc" (UID: "1d9f58fc-5928-4213-8e45-380962645bd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:12.175657 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:12.175566    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:12.175657 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:12.175620    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:12.175863 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:12.175707    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:12.175863 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:12.175839    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:13.175657 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:13.175627    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:13.176062 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:13.175745    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:14.175204 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:14.175024    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:14.175370 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:14.175025    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:14.175370 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:14.175314    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:14.175500 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:14.175370    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:15.179407 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:15.179376    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:15.179874 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:15.179483    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:16.175044 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.175021    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:16.175160 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:16.175120    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:16.175219 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.175180    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:16.175305 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:16.175286    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:16.442141 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.441840    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xxpbg" event={"ID":"4008895f-2b03-4540-96d8-773fea27114f","Type":"ContainerStarted","Data":"c459c670733237fdf427b101b2c99db46a0b3f570e1716f055e58f6e2d689395"}
Jul 28 17:17:16.449005 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.448640    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fz847" event={"ID":"0a5be7c1-fe99-43a9-acd3-912ea1a5c61e","Type":"ContainerStarted","Data":"8a500f3e77ee533103f1dc850bedb0faa3ca966a65c89fd4374db7201eb9a357"}
Jul 28 17:17:16.452251 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452186    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:17:16.452875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452537    2585 generic.go:356] "Generic (PLEG): container finished" podID="3a35d96c-75c0-4979-9470-4a4fc570ed7d" containerID="38f195d810d9b6402c5b21c18bbb1e06b3649e52cfaf366f189e04ba2decc9da" exitCode=1
Jul 28 17:17:16.452875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452584    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"7a2007fe38cf04c2c060d70de5aff281bf6fa142c0705b157c650259a6eeb742"}
Jul 28 17:17:16.452875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452657    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"8a1ba81d5410faad1edef96533d113feecadf0ed51e83dd378c746220a4c87d1"}
Jul 28 17:17:16.452875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"19ad3ef9217df1ceecff7cf1a9897c4a6a38cd0c1c05208bf1a60b6515f71595"}
Jul 28 17:17:16.452875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452740    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"dd8b9ef6b0d675078b5121783e01a11860ae5714b3d896deb9f5cd1791946f3a"}
Jul 28 17:17:16.452875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerDied","Data":"38f195d810d9b6402c5b21c18bbb1e06b3649e52cfaf366f189e04ba2decc9da"}
Jul 28 17:17:16.452875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.452822    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"a031ae0f19fcaab3717eae4106e21df801ef3a7c8312f3cb762c538374d9934b"}
Jul 28 17:17:16.454410 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.454390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6vs6" event={"ID":"1b95f9c4-e82e-4530-bfb1-740d8c4e8c22","Type":"ContainerStarted","Data":"977b2373892e6ac25ed3592c2251b6e79a2b2340225065ac7cc78a2ab147ee07"}
Jul 28 17:17:16.456221 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.456107    2585 generic.go:356] "Generic (PLEG): container finished" podID="789ac91acd8f458997b00ba308154756" containerID="afc5b3fb18de2276a480769d2c0362adbea00a8e3060eb208ac5f61a5e5c388b" exitCode=0
Jul 28 17:17:16.456496 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.456478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal" event={"ID":"789ac91acd8f458997b00ba308154756","Type":"ContainerDied","Data":"afc5b3fb18de2276a480769d2c0362adbea00a8e3060eb208ac5f61a5e5c388b"}
Jul 28 17:17:16.458305 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.458279    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gwcq" event={"ID":"07aa6d61-639a-4114-9f14-21ef6bfe55fd","Type":"ContainerStarted","Data":"7e233130c79819e0c192c1ad6a7904b5747124812de91b3721c8b4f50a3f5f61"}
Jul 28 17:17:16.459751 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.459731    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerStarted","Data":"ed8d237da1dac990a4e70971e7eb76330b7641b058f13d844daeae12e5947d7b"}
Jul 28 17:17:16.461440 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.461374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr" event={"ID":"10748634-0be3-4fd0-8789-da7427282953","Type":"ContainerStarted","Data":"02ee21cd8ae058aa0fa65768ae45414f2f243d8e34944e76ea3ef3f99c47f7ca"}
Jul 28 17:17:16.463271 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.463247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal" event={"ID":"0dd2e9a077ed553dd3893a28c3e87410","Type":"ContainerStarted","Data":"e8a3db64828b8821d6eabd18759f851c09c0ed421ac15eecea4a0b56069eed9c"}
Jul 28 17:17:16.465223 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.465198    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7nqzl" event={"ID":"c3e7698f-f10c-4566-9052-349b0610b252","Type":"ContainerStarted","Data":"bf872e5cdd2494d81dd1d0d126efc0b4756fcae7f1e31f57595545c5ff163dbd"}
Jul 28 17:17:16.467272 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.467234    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xxpbg" podStartSLOduration=2.187843865 podStartE2EDuration="21.46722336s" totalImagesPullingTime="19.279379495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:16.467055035 +0000 UTC m=+21.810647313" watchObservedRunningTime="2026-07-28 17:17:16.46722336 +0000 UTC m=+21.810815636"
Jul 28 17:17:16.504157 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.504099    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal" podStartSLOduration=21.504078342 podStartE2EDuration="21.504078342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:16.503099652 +0000 UTC m=+21.846691931" watchObservedRunningTime="2026-07-28 17:17:16.504078342 +0000 UTC m=+21.847670625"
Jul 28 17:17:16.544379 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.544333    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f6vs6" podStartSLOduration=1.599049515 podStartE2EDuration="21.5443152s" totalImagesPullingTime="19.945265685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:16.543877094 +0000 UTC m=+21.887469372" watchObservedRunningTime="2026-07-28 17:17:16.5443152 +0000 UTC m=+21.887907477"
Jul 28 17:17:16.566949 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.566898    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fz847" podStartSLOduration=2.2995276000000002 podStartE2EDuration="21.566883174s" totalImagesPullingTime="19.267355574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:16.566470109 +0000 UTC m=+21.910062385" watchObservedRunningTime="2026-07-28 17:17:16.566883174 +0000 UTC m=+21.910475451"
Jul 28 17:17:16.583767 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.583725    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8gwcq" podStartSLOduration=2.338695716 podStartE2EDuration="21.583710301s" totalImagesPullingTime="19.245014585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:16.58348419 +0000 UTC m=+21.927076463" watchObservedRunningTime="2026-07-28 17:17:16.583710301 +0000 UTC m=+21.927302577"
Jul 28 17:17:16.600937 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:16.600835    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7nqzl" podStartSLOduration=6.201030154 podStartE2EDuration="21.600806213s" totalImagesPullingTime="15.399776059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:16.599901209 +0000 UTC m=+21.943493485" watchObservedRunningTime="2026-07-28 17:17:16.600806213 +0000 UTC m=+21.944398525"
Jul 28 17:17:17.175022 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.174952    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:17.175163 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:17.175066    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:17.235663 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.235628    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:17:17.236378 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.236356    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:17:17.470273 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.470184    2585 generic.go:356] "Generic (PLEG): container finished" podID="91aee170-ad83-440e-8544-ccc56a8f216a" containerID="ed8d237da1dac990a4e70971e7eb76330b7641b058f13d844daeae12e5947d7b" exitCode=0
Jul 28 17:17:17.470273 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.470255    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerDied","Data":"ed8d237da1dac990a4e70971e7eb76330b7641b058f13d844daeae12e5947d7b"}
Jul 28 17:17:17.471724 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.471698    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqqgn" event={"ID":"e0ab3143-f0ea-44c3-b554-b78e9d8849b2","Type":"ContainerStarted","Data":"878bf8128839b7c74735629a82883f300dd6d58de14a822180308b1be06ec700"}
Jul 28 17:17:17.473305 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.473271    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal" event={"ID":"789ac91acd8f458997b00ba308154756","Type":"ContainerStarted","Data":"10c5aa154c46d10e5acb965d86201b814f0f1d1e1e2ed5324d9ae6f855c05824"}
Jul 28 17:17:17.473488 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.473470    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:17:17.474113 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.474095    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fz847"
Jul 28 17:17:17.532020 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.531964    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gqqgn" podStartSLOduration=3.292545415 podStartE2EDuration="22.531949104s" totalImagesPullingTime="19.239403689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:17.531398093 +0000 UTC m=+22.874990371" watchObservedRunningTime="2026-07-28 17:17:17.531949104 +0000 UTC m=+22.875541380"
Jul 28 17:17:17.550278 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.550231    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-20.ec2.internal" podStartSLOduration=22.550217647 podStartE2EDuration="22.550217647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:17.550102543 +0000 UTC m=+22.893694820" watchObservedRunningTime="2026-07-28 17:17:17.550217647 +0000 UTC m=+22.893809926"
Jul 28 17:17:17.769593 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:17.769571    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:17:18.068318 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:18.068161    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:17:17.769588434Z","UUID":"e00edb71-53c6-42e0-8872-a83fd57c0555","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:17:18.069883 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:18.069863    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:17:18.070002 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:18.069899    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:17:18.175236 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:18.175203    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:18.175421 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:18.175206    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:18.175421 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:18.175352    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:18.175546 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:18.175416    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:18.478474 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:18.478439    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr" event={"ID":"10748634-0be3-4fd0-8789-da7427282953","Type":"ContainerStarted","Data":"9c5926f3b8d73514a3f6a002ef9d54fca8d0f3ace2789a858724157febb9aac4"}
Jul 28 17:17:19.174939 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:19.174910    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:19.175136 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:19.175046    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:19.484056 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:19.483988    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:17:19.484496 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:19.484436    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"a951fad59644907b84e86fcdc85a38c42c8c4b84db8d314d27dd53920d7a6e2c"}
Jul 28 17:17:19.486713 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:19.486667    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr" event={"ID":"10748634-0be3-4fd0-8789-da7427282953","Type":"ContainerStarted","Data":"8ef6217b0e58d3c1f9e88deb6323aa5fc8e58351fcb79f6bf08f965c1b16203c"}
Jul 28 17:17:19.509927 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:19.509881    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9wttr" podStartSLOduration=2.683080013 podStartE2EDuration="24.509863499s" totalImagesPullingTime="21.826783486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:19.509731023 +0000 UTC m=+24.853323299" watchObservedRunningTime="2026-07-28 17:17:19.509863499 +0000 UTC m=+24.853455777"
Jul 28 17:17:20.174921 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:20.174885    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:20.175102 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:20.174897    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:20.175102 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:20.174985    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:20.175216 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:20.175118    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:21.175193 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:21.175165    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:21.175617 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:21.175289    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:22.175798 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.175569    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:22.176479 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.175599    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:22.176479 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:22.175866    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:22.176479 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:22.175897    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:22.495654 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.495581    2585 generic.go:356] "Generic (PLEG): container finished" podID="91aee170-ad83-440e-8544-ccc56a8f216a" containerID="43e28be8596eceab1d1fdd638f33cd5336ac5415439f42a5e78bf015b406e908" exitCode=0
Jul 28 17:17:22.495840 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.495671    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerDied","Data":"43e28be8596eceab1d1fdd638f33cd5336ac5415439f42a5e78bf015b406e908"}
Jul 28 17:17:22.499114 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.499101    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:17:22.499435 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.499415    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"4432be084761464641ed6d447827868f6d15f1b3bfbef8cbc5e57f7670a5cab9"}
Jul 28 17:17:22.499747 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.499726    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:17:22.499843 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.499793    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:17:22.499843 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.499822    2585 scope.go:117] "RemoveContainer" containerID="38f195d810d9b6402c5b21c18bbb1e06b3649e52cfaf366f189e04ba2decc9da"
Jul 28 17:17:22.499843 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.499832    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:17:22.515595 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.515565    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:17:22.518745 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:22.518725    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:17:23.175256 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.175108    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:23.175446 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:23.175381    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:23.504849 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.504811    2585 generic.go:356] "Generic (PLEG): container finished" podID="91aee170-ad83-440e-8544-ccc56a8f216a" containerID="9cfd8b594fe53e1e979ce03733bc2845ac82e2cc0bbaf1e614b2861cc4cfb4be" exitCode=0
Jul 28 17:17:23.505227 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.504899    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerDied","Data":"9cfd8b594fe53e1e979ce03733bc2845ac82e2cc0bbaf1e614b2861cc4cfb4be"}
Jul 28 17:17:23.509165 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.509146    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:17:23.509516 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.509491    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" event={"ID":"3a35d96c-75c0-4979-9470-4a4fc570ed7d","Type":"ContainerStarted","Data":"814bb371ff4bb1c922d059576a9609f20e74302ff68f23c4be389555ff8d1f45"}
Jul 28 17:17:23.569667 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.569623    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll" podStartSLOduration=9.147452202 podStartE2EDuration="28.569608399s" totalImagesPullingTime="19.422156197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:23.569512932 +0000 UTC m=+28.913105208" watchObservedRunningTime="2026-07-28 17:17:23.569608399 +0000 UTC m=+28.913200718"
Jul 28 17:17:23.746970 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.746898    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-27fcp"]
Jul 28 17:17:23.747085 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.746986    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:23.747085 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:23.747067    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:23.752217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.752189    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gsqc"]
Jul 28 17:17:23.752377 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.752283    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ng94v"]
Jul 28 17:17:23.752463 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.752387    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:23.752542 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:23.752519    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:23.752636 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:23.752622    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:23.752735 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:23.752719    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:25.175844 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:25.175814    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:25.176236 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:25.175899    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:25.176236 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:25.175977    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:25.176236 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:25.176096    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:25.515798 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:25.515722    2585 generic.go:356] "Generic (PLEG): container finished" podID="91aee170-ad83-440e-8544-ccc56a8f216a" containerID="b2cc19cd62a7846f4d77f2bfa50eb824b953eb7f20a5002a8a10e9ca76296089" exitCode=0
Jul 28 17:17:25.515798 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:25.515781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerDied","Data":"b2cc19cd62a7846f4d77f2bfa50eb824b953eb7f20a5002a8a10e9ca76296089"}
Jul 28 17:17:26.175505 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:26.175476    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:26.175654 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:26.175605    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ng94v" podUID="0a3ff851-6c38-46db-b020-57ae09a57aa3"
Jul 28 17:17:27.174850 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:27.174650    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:27.175287 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:27.174650    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:27.175287 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.175044    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27fcp" podUID="8f4bfba2-733c-453a-a03d-02d4994797d4"
Jul 28 17:17:27.175287 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.174924    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gsqc" podUID="1d9f58fc-5928-4213-8e45-380962645bd1"
Jul 28 17:17:27.859982 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:27.859946    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:27.860146 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:27.860004    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:27.860146 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.860115    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:27.860146 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.860121    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:27.860316 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.860169    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret podName:8f4bfba2-733c-453a-a03d-02d4994797d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.860155477 +0000 UTC m=+65.203747732 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret") pod "global-pull-secret-syncer-27fcp" (UID: "8f4bfba2-733c-453a-a03d-02d4994797d4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:27.860316 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.860183    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.86017675 +0000 UTC m=+65.203769005 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:27.961249 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:27.961212    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:27.961440 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.961362    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:27.961440 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.961382    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:27.961440 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.961395    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ktt2f for pod openshift-network-diagnostics/network-check-target-4gsqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:27.961612 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:27.961453    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f podName:1d9f58fc-5928-4213-8e45-380962645bd1 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.961436941 +0000 UTC m=+65.305029199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktt2f" (UniqueName: "kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f") pod "network-check-target-4gsqc" (UID: "1d9f58fc-5928-4213-8e45-380962645bd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:27.982570 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:27.982493    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-20.ec2.internal" event="NodeReady"
Jul 28 17:17:27.982698 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:27.982646    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:17:28.054848 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.054815    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cqcfv"]
Jul 28 17:17:28.058415 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.058390    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.066170 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.066145    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:17:28.066272 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.066178    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sd8qz"]
Jul 28 17:17:28.066272 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.066240    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:17:28.067334 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.067316    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c9wr5\""
Jul 28 17:17:28.069923 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.069905    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:28.075986 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.075956    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 28 17:17:28.076099 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.075977    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 28 17:17:28.076099 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.076075    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"canary-serving-cert\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret"
Jul 28 17:17:28.076286 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.076157    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-dzv2b\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dzv2b\"" type="*v1.Secret"
Jul 28 17:17:28.083771 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.083750    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cqcfv"]
Jul 28 17:17:28.088969 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.088932    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sd8qz"]
Jul 28 17:17:28.163280 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.163246    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fbr\" (UniqueName: \"kubernetes.io/projected/e537ee6c-b73a-46e2-8e85-9d4dce87ea88-kube-api-access-g4fbr\") pod \"ingress-canary-sd8qz\" (UID: \"e537ee6c-b73a-46e2-8e85-9d4dce87ea88\") " pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:28.163434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.163312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.163434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.163339    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8ds\" (UniqueName: \"kubernetes.io/projected/def621fc-443a-4f17-8c74-bbe3605a4c89-kube-api-access-bl8ds\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.163434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.163365    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def621fc-443a-4f17-8c74-bbe3605a4c89-tmp-dir\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.163434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.163403    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e537ee6c-b73a-46e2-8e85-9d4dce87ea88-cert\") pod \"ingress-canary-sd8qz\" (UID: \"e537ee6c-b73a-46e2-8e85-9d4dce87ea88\") " pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:28.163434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.163429    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/def621fc-443a-4f17-8c74-bbe3605a4c89-config-volume\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.175440 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.175403    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:28.178242 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.178221    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:17:28.178345 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.178331    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2k2q\""
Jul 28 17:17:28.264334 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264248    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e537ee6c-b73a-46e2-8e85-9d4dce87ea88-cert\") pod \"ingress-canary-sd8qz\" (UID: \"e537ee6c-b73a-46e2-8e85-9d4dce87ea88\") " pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:28.264334 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/def621fc-443a-4f17-8c74-bbe3605a4c89-config-volume\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.264535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264374    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fbr\" (UniqueName: \"kubernetes.io/projected/e537ee6c-b73a-46e2-8e85-9d4dce87ea88-kube-api-access-g4fbr\") pod \"ingress-canary-sd8qz\" (UID: \"e537ee6c-b73a-46e2-8e85-9d4dce87ea88\") " pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:28.264535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264434    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.264535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8ds\" (UniqueName: \"kubernetes.io/projected/def621fc-443a-4f17-8c74-bbe3605a4c89-kube-api-access-bl8ds\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.264535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264497    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def621fc-443a-4f17-8c74-bbe3605a4c89-tmp-dir\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.264766 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.264605    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:28.264766 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.264667    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls podName:def621fc-443a-4f17-8c74-bbe3605a4c89 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:28.764648387 +0000 UTC m=+34.108240642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls") pod "dns-default-cqcfv" (UID: "def621fc-443a-4f17-8c74-bbe3605a4c89") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:28.264888 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264840    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def621fc-443a-4f17-8c74-bbe3605a4c89-tmp-dir\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.265001 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.264982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/def621fc-443a-4f17-8c74-bbe3605a4c89-config-volume\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.275813 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.275792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8ds\" (UniqueName: \"kubernetes.io/projected/def621fc-443a-4f17-8c74-bbe3605a4c89-kube-api-access-bl8ds\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.767643 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.767601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:28.767838 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.767741    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:28.767897 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:28.767850    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls podName:def621fc-443a-4f17-8c74-bbe3605a4c89 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:29.76782794 +0000 UTC m=+35.111420213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls") pod "dns-default-cqcfv" (UID: "def621fc-443a-4f17-8c74-bbe3605a4c89") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:28.906450 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.906422    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:17:28.917521 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:28.917494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e537ee6c-b73a-46e2-8e85-9d4dce87ea88-cert\") pod \"ingress-canary-sd8qz\" (UID: \"e537ee6c-b73a-46e2-8e85-9d4dce87ea88\") " pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:29.124007 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.123935    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:17:29.178411 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.178383    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:17:29.178411 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.178395    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:29.181340 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.181318    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wqrl8\""
Jul 28 17:17:29.181451 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.181321    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:17:29.182499 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.182434    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:17:29.182499 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.182486    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:17:29.249744 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.249714    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dzv2b\""
Jul 28 17:17:29.440141 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.440066    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:17:29.445814 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.445785    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fbr\" (UniqueName: \"kubernetes.io/projected/e537ee6c-b73a-46e2-8e85-9d4dce87ea88-kube-api-access-g4fbr\") pod \"ingress-canary-sd8qz\" (UID: \"e537ee6c-b73a-46e2-8e85-9d4dce87ea88\") " pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:29.580134 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.580105    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sd8qz"
Jul 28 17:17:29.775774 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.775616    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:29.775878 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:29.775756    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:29.775912 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:29.775895    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls podName:def621fc-443a-4f17-8c74-bbe3605a4c89 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:31.775881262 +0000 UTC m=+37.119473517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls") pod "dns-default-cqcfv" (UID: "def621fc-443a-4f17-8c74-bbe3605a4c89") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:29.783919 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:29.783893    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sd8qz"]
Jul 28 17:17:29.786820 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:17:29.786795    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode537ee6c_b73a_46e2_8e85_9d4dce87ea88.slice/crio-432acbd2cc2241bb893177e5e8eba75ea2fe0e7bca3ea6058ba3175a2b4f40e8 WatchSource:0}: Error finding container 432acbd2cc2241bb893177e5e8eba75ea2fe0e7bca3ea6058ba3175a2b4f40e8: Status 404 returned error can't find the container with id 432acbd2cc2241bb893177e5e8eba75ea2fe0e7bca3ea6058ba3175a2b4f40e8
Jul 28 17:17:30.532439 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:30.532404    2585 generic.go:356] "Generic (PLEG): container finished" podID="91aee170-ad83-440e-8544-ccc56a8f216a" containerID="53895c510412f8e24403f1986f23c9ed13680ae22e8f062065e7c9a15c188509" exitCode=0
Jul 28 17:17:30.533208 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:30.532487    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerDied","Data":"53895c510412f8e24403f1986f23c9ed13680ae22e8f062065e7c9a15c188509"}
Jul 28 17:17:30.533737 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:30.533708    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sd8qz" event={"ID":"e537ee6c-b73a-46e2-8e85-9d4dce87ea88","Type":"ContainerStarted","Data":"432acbd2cc2241bb893177e5e8eba75ea2fe0e7bca3ea6058ba3175a2b4f40e8"}
Jul 28 17:17:31.540700 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:31.540666    2585 generic.go:356] "Generic (PLEG): container finished" podID="91aee170-ad83-440e-8544-ccc56a8f216a" containerID="aaff50631c1d2f3f4ffecdee9940be1a18a01fd4d17b67cc27f32af2abf1e854" exitCode=0
Jul 28 17:17:31.540700 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:31.540708    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerDied","Data":"aaff50631c1d2f3f4ffecdee9940be1a18a01fd4d17b67cc27f32af2abf1e854"}
Jul 28 17:17:31.790211 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:31.790174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:31.790372 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:31.790313    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:31.790429 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:31.790389    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls podName:def621fc-443a-4f17-8c74-bbe3605a4c89 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:35.790369275 +0000 UTC m=+41.133961532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls") pod "dns-default-cqcfv" (UID: "def621fc-443a-4f17-8c74-bbe3605a4c89") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:32.546282 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:32.546248    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" event={"ID":"91aee170-ad83-440e-8544-ccc56a8f216a","Type":"ContainerStarted","Data":"459b7c644264ee3f0fea6fb6ead7ff7aa00619962b97849bb826ed5acecc1347"}
Jul 28 17:17:32.547602 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:32.547583    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sd8qz" event={"ID":"e537ee6c-b73a-46e2-8e85-9d4dce87ea88","Type":"ContainerStarted","Data":"de31d46ecb9e0bb06e7aadd3be94d476b8a87485bdc2f602bce9e2bd9edf44f3"}
Jul 28 17:17:32.572835 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:32.572780    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8xzdj" podStartSLOduration=8.02637311 podStartE2EDuration="37.572764559s" totalImagesPullingTime="29.546391449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:32.569501674 +0000 UTC m=+37.913093952" watchObservedRunningTime="2026-07-28 17:17:32.572764559 +0000 UTC m=+37.916356837"
Jul 28 17:17:32.585817 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:32.585775    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sd8qz" podStartSLOduration=2.132880705 podStartE2EDuration="4.585759714s" totalImagesPullingTime="2.452879009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:32.585318992 +0000 UTC m=+37.928911269" watchObservedRunningTime="2026-07-28 17:17:32.585759714 +0000 UTC m=+37.929351993"
Jul 28 17:17:35.819007 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:35.818968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:35.819466 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:35.819091    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:35.819466 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:35.819142    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls podName:def621fc-443a-4f17-8c74-bbe3605a4c89 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:43.819127985 +0000 UTC m=+49.162720239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls") pod "dns-default-cqcfv" (UID: "def621fc-443a-4f17-8c74-bbe3605a4c89") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:43.737961 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.737925    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"]
Jul 28 17:17:43.781849 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.781823    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"]
Jul 28 17:17:43.781983 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.781916    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.786071 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.786049    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 17:17:43.786605 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.786584    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 17:17:43.786694 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.786587    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:43.786694 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.786587    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:43.786694 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.786611    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5z5dz\""
Jul 28 17:17:43.844720 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.844696    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66dc97f78c-hsl5f"]
Jul 28 17:17:43.857233 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.857210    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:43.862966 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.862941    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66dc97f78c-hsl5f"]
Jul 28 17:17:43.863481 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.863467    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:17:43.863651 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.863628    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:17:43.863747 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.863706    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 17:17:43.864038 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.863999    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 17:17:43.864038 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.864014    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 17:17:43.864334 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.864101    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xj5xx\""
Jul 28 17:17:43.864334 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.864222    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 17:17:43.872802 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.872784    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:43.872886 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.872823    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxx4w\" (UniqueName: \"kubernetes.io/projected/76fd56c1-4eba-4445-a7b5-a75f304f150b-kube-api-access-sxx4w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.872886 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.872863    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76fd56c1-4eba-4445-a7b5-a75f304f150b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.872886 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.872881    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76fd56c1-4eba-4445-a7b5-a75f304f150b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.873004 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:43.872931    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:43.873004 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:43.872980    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls podName:def621fc-443a-4f17-8c74-bbe3605a4c89 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.872967027 +0000 UTC m=+65.216559282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls") pod "dns-default-cqcfv" (UID: "def621fc-443a-4f17-8c74-bbe3605a4c89") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:43.957919 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.957895    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"]
Jul 28 17:17:43.963279 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.963263    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:43.965820 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.965797    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 17:17:43.965933 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.965833    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dkxfr\""
Jul 28 17:17:43.965933 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.965876    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 17:17:43.966116 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.966101    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:43.966116 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.966110    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:43.972755 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.972735    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"]
Jul 28 17:17:43.974160 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974143    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-default-certificate\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:43.974227 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974167    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:43.974227 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974192    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxx4w\" (UniqueName: \"kubernetes.io/projected/76fd56c1-4eba-4445-a7b5-a75f304f150b-kube-api-access-sxx4w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.974227 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-stats-auth\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:43.974361 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974260    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsx9p\" (UniqueName: \"kubernetes.io/projected/3cc057ee-b81c-4977-ba8a-b8a2f478729b-kube-api-access-dsx9p\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:43.974414 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974371    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76fd56c1-4eba-4445-a7b5-a75f304f150b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.974466 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974410    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76fd56c1-4eba-4445-a7b5-a75f304f150b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.974466 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.974441    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:43.975086 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.975059    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76fd56c1-4eba-4445-a7b5-a75f304f150b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.978370 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.978345    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76fd56c1-4eba-4445-a7b5-a75f304f150b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:43.986868 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:43.986847    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxx4w\" (UniqueName: \"kubernetes.io/projected/76fd56c1-4eba-4445-a7b5-a75f304f150b-kube-api-access-sxx4w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf\" (UID: \"76fd56c1-4eba-4445-a7b5-a75f304f150b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:44.074889 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.074860    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ns4j\" (UniqueName: \"kubernetes.io/projected/28a356b3-2909-4397-8373-0eaf6ba7a5e5-kube-api-access-8ns4j\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.075064 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.074897    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-default-certificate\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.075064 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.074914    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.075064 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.074936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-stats-auth\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.075064 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.074979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsx9p\" (UniqueName: \"kubernetes.io/projected/3cc057ee-b81c-4977-ba8a-b8a2f478729b-kube-api-access-dsx9p\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.075254 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:44.075079    2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:44.075254 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.075086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/28a356b3-2909-4397-8373-0eaf6ba7a5e5-serving-cert\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.075254 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:44.075144    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:44.575124075 +0000 UTC m=+49.918716334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : secret "router-metrics-certs-default" not found
Jul 28 17:17:44.075254 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.075189    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/28a356b3-2909-4397-8373-0eaf6ba7a5e5-config\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.075254 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.075222    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.075481 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:44.075343    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:44.575327127 +0000 UTC m=+49.918919384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:44.077400 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.077379    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-default-certificate\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.077501 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.077455    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-stats-auth\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.084660 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.084644    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsx9p\" (UniqueName: \"kubernetes.io/projected/3cc057ee-b81c-4977-ba8a-b8a2f478729b-kube-api-access-dsx9p\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.101637 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.101618    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"
Jul 28 17:17:44.176281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.176252    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/28a356b3-2909-4397-8373-0eaf6ba7a5e5-serving-cert\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.176424 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.176306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/28a356b3-2909-4397-8373-0eaf6ba7a5e5-config\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.176424 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.176391    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ns4j\" (UniqueName: \"kubernetes.io/projected/28a356b3-2909-4397-8373-0eaf6ba7a5e5-kube-api-access-8ns4j\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.176986 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.176955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/28a356b3-2909-4397-8373-0eaf6ba7a5e5-config\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.178504 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.178482    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/28a356b3-2909-4397-8373-0eaf6ba7a5e5-serving-cert\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.189535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.189507    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ns4j\" (UniqueName: \"kubernetes.io/projected/28a356b3-2909-4397-8373-0eaf6ba7a5e5-kube-api-access-8ns4j\") pod \"service-ca-operator-5d6c75f84f-d65fd\" (UID: \"28a356b3-2909-4397-8373-0eaf6ba7a5e5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.253687 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.253654    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf"]
Jul 28 17:17:44.256705 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:17:44.256663    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fd56c1_4eba_4445_a7b5_a75f304f150b.slice/crio-4598dc5de066b6b85c1db48c9df455b667afa843515b6adba642f90457b1ac14 WatchSource:0}: Error finding container 4598dc5de066b6b85c1db48c9df455b667afa843515b6adba642f90457b1ac14: Status 404 returned error can't find the container with id 4598dc5de066b6b85c1db48c9df455b667afa843515b6adba642f90457b1ac14
Jul 28 17:17:44.271643 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.271619    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"
Jul 28 17:17:44.399826 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.399800    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd"]
Jul 28 17:17:44.402663 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:17:44.402626    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a356b3_2909_4397_8373_0eaf6ba7a5e5.slice/crio-4bd87dcbf1e1af113f87b75a818e5a37a3167a798eb373f21dd222ef442b289a WatchSource:0}: Error finding container 4bd87dcbf1e1af113f87b75a818e5a37a3167a798eb373f21dd222ef442b289a: Status 404 returned error can't find the container with id 4bd87dcbf1e1af113f87b75a818e5a37a3167a798eb373f21dd222ef442b289a
Jul 28 17:17:44.575987 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.575948    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd" event={"ID":"28a356b3-2909-4397-8373-0eaf6ba7a5e5","Type":"ContainerStarted","Data":"4bd87dcbf1e1af113f87b75a818e5a37a3167a798eb373f21dd222ef442b289a"}
Jul 28 17:17:44.576960 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.576939    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf" event={"ID":"76fd56c1-4eba-4445-a7b5-a75f304f150b","Type":"ContainerStarted","Data":"4598dc5de066b6b85c1db48c9df455b667afa843515b6adba642f90457b1ac14"}
Jul 28 17:17:44.579288 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.579243    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.579345 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:44.579302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:44.579399 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:44.579388    2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:44.579433 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:44.579399    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.579382254 +0000 UTC m=+50.922974509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:44.579433 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:44.579430    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.579417226 +0000 UTC m=+50.923009480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : secret "router-metrics-certs-default" not found
Jul 28 17:17:45.588215 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:45.588182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:45.588722 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:45.588254    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:45.588722 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:45.588357    2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:45.588722 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:45.588386    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:47.588369452 +0000 UTC m=+52.931961706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:45.588722 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:45.588419    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:47.588401843 +0000 UTC m=+52.931994111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : secret "router-metrics-certs-default" not found
Jul 28 17:17:47.586280 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:47.586237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd" event={"ID":"28a356b3-2909-4397-8373-0eaf6ba7a5e5","Type":"ContainerStarted","Data":"b4e563d054ed158434d1d45c310badfd2afb55e3c58feef12631c92995058852"}
Jul 28 17:17:47.587739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:47.587709    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf" event={"ID":"76fd56c1-4eba-4445-a7b5-a75f304f150b","Type":"ContainerStarted","Data":"9f2efb1da22f1af8edce6dffd505d3bca67d559d46592056c9e57e4b2166d627"}
Jul 28 17:17:47.606286 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:47.606253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:47.606413 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:47.606302    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd" podStartSLOduration=1.803020166 podStartE2EDuration="4.606287772s" totalImagesPullingTime="2.803267606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:47.604826143 +0000 UTC m=+52.948418423" watchObservedRunningTime="2026-07-28 17:17:47.606287772 +0000 UTC m=+52.949880051"
Jul 28 17:17:47.606413 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:47.606395    2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:47.606529 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:47.606458    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:51.606440369 +0000 UTC m=+56.950032628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : secret "router-metrics-certs-default" not found
Jul 28 17:17:47.606529 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:47.606511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:47.606845 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:47.606824    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:51.606805593 +0000 UTC m=+56.950397867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:47.621439 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:47.621367    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf" podStartSLOduration=1.6666558500000002 podStartE2EDuration="4.621350328s" totalImagesPullingTime="2.954694478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:47.621010299 +0000 UTC m=+52.964602576" watchObservedRunningTime="2026-07-28 17:17:47.621350328 +0000 UTC m=+52.964942606"
Jul 28 17:17:49.297139 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.297107    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bttvz"]
Jul 28 17:17:49.299743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.299726    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.302640 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.302614    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:17:49.303861 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.303841    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:17:49.303967 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.303903    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hbmbv\""
Jul 28 17:17:49.303967 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.303904    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:17:49.303967 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.303959    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:17:49.314758 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.314734    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bttvz"]
Jul 28 17:17:49.320147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.320120    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bb9c484-525b-430e-bee3-69208d19d269-data-volume\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.320258 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.320179    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bb9c484-525b-430e-bee3-69208d19d269-crio-socket\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.320258 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.320202    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.320258 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.320233    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkw6\" (UniqueName: \"kubernetes.io/projected/7bb9c484-525b-430e-bee3-69208d19d269-kube-api-access-sdkw6\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.320398 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.320266    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.420766 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.420719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bb9c484-525b-430e-bee3-69208d19d269-crio-socket\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.420933 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.420837    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bb9c484-525b-430e-bee3-69208d19d269-crio-socket\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.420933 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.420859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.420933 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.420890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkw6\" (UniqueName: \"kubernetes.io/projected/7bb9c484-525b-430e-bee3-69208d19d269-kube-api-access-sdkw6\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.420933 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.420927    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.421062 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.420977    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:49.421062 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.420930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.421062 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.420979    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:49.920963599 +0000 UTC m=+55.264555854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.421062 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.421057    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:49.921042245 +0000 UTC m=+55.264634510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:49.421246 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.421155    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bb9c484-525b-430e-bee3-69208d19d269-data-volume\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.421424 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.421408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bb9c484-525b-430e-bee3-69208d19d269-data-volume\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.430456 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.430434    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkw6\" (UniqueName: \"kubernetes.io/projected/7bb9c484-525b-430e-bee3-69208d19d269-kube-api-access-sdkw6\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.755981 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.755896    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"]
Jul 28 17:17:49.757907 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.757889    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"
Jul 28 17:17:49.761314 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.761292    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cc94t\""
Jul 28 17:17:49.761430 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.761292    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:49.761606 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.761591    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 17:17:49.769824 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.769802    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"]
Jul 28 17:17:49.824636 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.824596    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdz7\" (UniqueName: \"kubernetes.io/projected/4eb017f2-945a-4a40-a843-d1ae387db67e-kube-api-access-wzdz7\") pod \"migrator-6865c9967-xrcn8\" (UID: \"4eb017f2-945a-4a40-a843-d1ae387db67e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"
Jul 28 17:17:49.925406 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.925371    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.925585 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.925424    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:49.925585 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.925474    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdz7\" (UniqueName: \"kubernetes.io/projected/4eb017f2-945a-4a40-a843-d1ae387db67e-kube-api-access-wzdz7\") pod \"migrator-6865c9967-xrcn8\" (UID: \"4eb017f2-945a-4a40-a843-d1ae387db67e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"
Jul 28 17:17:49.925585 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.925534    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.925749 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.925582    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:49.925749 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.925633    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:50.925611523 +0000 UTC m=+56.269203792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.925749 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:49.925655    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:50.925645009 +0000 UTC m=+56.269237271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:49.934355 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:49.934327    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdz7\" (UniqueName: \"kubernetes.io/projected/4eb017f2-945a-4a40-a843-d1ae387db67e-kube-api-access-wzdz7\") pod \"migrator-6865c9967-xrcn8\" (UID: \"4eb017f2-945a-4a40-a843-d1ae387db67e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"
Jul 28 17:17:50.067147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:50.067122    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"
Jul 28 17:17:50.188413 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:50.188386    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8"]
Jul 28 17:17:50.190337 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:17:50.190302    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb017f2_945a_4a40_a843_d1ae387db67e.slice/crio-8509553cb252b9358766c44ea57922b8c230874f816cd2f832671c40bdb36c48 WatchSource:0}: Error finding container 8509553cb252b9358766c44ea57922b8c230874f816cd2f832671c40bdb36c48: Status 404 returned error can't find the container with id 8509553cb252b9358766c44ea57922b8c230874f816cd2f832671c40bdb36c48
Jul 28 17:17:50.596455 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:50.596423    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8" event={"ID":"4eb017f2-945a-4a40-a843-d1ae387db67e","Type":"ContainerStarted","Data":"8509553cb252b9358766c44ea57922b8c230874f816cd2f832671c40bdb36c48"}
Jul 28 17:17:50.932707 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:50.932621    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:50.932707 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:50.932684    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:50.932901 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:50.932785    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:50.932901 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:50.932823    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:50.932901 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:50.932854    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:52.93283712 +0000 UTC m=+58.276429393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:50.932901 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:50.932877    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:52.932864568 +0000 UTC m=+58.276456826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:51.323261 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:51.323205    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gwcq_07aa6d61-639a-4114-9f14-21ef6bfe55fd/dns-node-resolver/0.log"
Jul 28 17:17:51.601386 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:51.601359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8" event={"ID":"4eb017f2-945a-4a40-a843-d1ae387db67e","Type":"ContainerStarted","Data":"31d619760d08695fab33f3b675cd2b12eee8ea5ec97868de8e6f972b96a6523a"}
Jul 28 17:17:51.601721 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:51.601406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8" event={"ID":"4eb017f2-945a-4a40-a843-d1ae387db67e","Type":"ContainerStarted","Data":"ce619003d92664fda1abb96e31802b76fe88cb083731b9e42fe36d37ee3fb4be"}
Jul 28 17:17:51.619972 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:51.619900    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xrcn8" podStartSLOduration=1.340327322 podStartE2EDuration="2.619885773s" totalImagesPullingTime="1.279558451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:51.61927628 +0000 UTC m=+56.962868558" watchObservedRunningTime="2026-07-28 17:17:51.619885773 +0000 UTC m=+56.963478049"
Jul 28 17:17:51.637859 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:51.637831    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:51.637975 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:51.637931    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:51.638016 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:51.637974    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.637958421 +0000 UTC m=+64.981550680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:51.638057 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:51.638038    2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:51.638101 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:51.638088    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.638074988 +0000 UTC m=+64.981667253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : secret "router-metrics-certs-default" not found
Jul 28 17:17:52.122654 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:52.122618    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7nqzl_c3e7698f-f10c-4566-9052-349b0610b252/node-ca/0.log"
Jul 28 17:17:52.947604 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:52.947565    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:52.948009 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:52.947631    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:52.948009 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:52.947649    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:52.948009 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:52.947712    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:56.947697444 +0000 UTC m=+62.291289702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:52.948009 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:52.947745    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:52.948009 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:52.947798    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:56.94778515 +0000 UTC m=+62.291377405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:53.321537 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:53.321504    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sd8qz_e537ee6c-b73a-46e2-8e85-9d4dce87ea88/serve-healthcheck-canary/0.log"
Jul 28 17:17:54.531529 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:54.531498    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxrll"
Jul 28 17:17:56.979714 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:56.979674    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:56.980104 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:56.979801    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:56.980104 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:56.979851    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:17:56.980104 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:56.979875    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:04.979860215 +0000 UTC m=+70.323452473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:56.980104 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:56.979946    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:56.980104 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:56.979997    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:04.979985796 +0000 UTC m=+70.323578050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:59.700694 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.700654    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:59.701099 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.700733    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:17:59.701099 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:59.700804    2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:59.701099 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:59.700873    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:18:15.70085879 +0000 UTC m=+81.044451044 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : secret "router-metrics-certs-default" not found
Jul 28 17:17:59.701099 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:59.700911    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle podName:3cc057ee-b81c-4977-ba8a-b8a2f478729b nodeName:}" failed. No retries permitted until 2026-07-28 17:18:15.70088702 +0000 UTC m=+81.044479286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle") pod "router-default-66dc97f78c-hsl5f" (UID: "3cc057ee-b81c-4977-ba8a-b8a2f478729b") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:59.903087 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.903052    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:17:59.903252 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.903131    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:17:59.903252 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.903149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:59.905347 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.905324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/def621fc-443a-4f17-8c74-bbe3605a4c89-metrics-tls\") pod \"dns-default-cqcfv\" (UID: \"def621fc-443a-4f17-8c74-bbe3605a4c89\") " pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:17:59.905997 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.905977    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:17:59.906115 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.906096    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:17:59.914156 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:59.914135    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:17:59.914252 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:17:59.914190    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs podName:0a3ff851-6c38-46db-b020-57ae09a57aa3 nodeName:}" failed. No retries permitted until 2026-07-28 17:19:03.914176003 +0000 UTC m=+129.257768261 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs") pod "network-metrics-daemon-ng94v" (UID: "0a3ff851-6c38-46db-b020-57ae09a57aa3") : secret "metrics-daemon-secret" not found
Jul 28 17:17:59.916648 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:17:59.916629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f4bfba2-733c-453a-a03d-02d4994797d4-original-pull-secret\") pod \"global-pull-secret-syncer-27fcp\" (UID: \"8f4bfba2-733c-453a-a03d-02d4994797d4\") " pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:18:00.003512 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.003421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:18:00.006300 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.006280    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:18:00.017145 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.017121    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:18:00.027717 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.027692    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktt2f\" (UniqueName: \"kubernetes.io/projected/1d9f58fc-5928-4213-8e45-380962645bd1-kube-api-access-ktt2f\") pod \"network-check-target-4gsqc\" (UID: \"1d9f58fc-5928-4213-8e45-380962645bd1\") " pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:18:00.091880 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.091849    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wqrl8\""
Jul 28 17:18:00.097330 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.097312    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27fcp"
Jul 28 17:18:00.099535 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.099517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:18:00.172944 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.172914    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c9wr5\""
Jul 28 17:18:00.180451 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.180210    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:18:00.227364 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.227334    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-27fcp"]
Jul 28 17:18:00.230836 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:00.230805    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4bfba2_733c_453a_a03d_02d4994797d4.slice/crio-3df3d1f1e739e89f7034887317675795a84edc7876e87921ab3450e4766392b3 WatchSource:0}: Error finding container 3df3d1f1e739e89f7034887317675795a84edc7876e87921ab3450e4766392b3: Status 404 returned error can't find the container with id 3df3d1f1e739e89f7034887317675795a84edc7876e87921ab3450e4766392b3
Jul 28 17:18:00.251632 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.251609    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gsqc"]
Jul 28 17:18:00.253671 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:00.253606    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9f58fc_5928_4213_8e45_380962645bd1.slice/crio-5dfa3cd1c86435951d4d75e584811be86203d11fc63aa90e229197376de2f9cb WatchSource:0}: Error finding container 5dfa3cd1c86435951d4d75e584811be86203d11fc63aa90e229197376de2f9cb: Status 404 returned error can't find the container with id 5dfa3cd1c86435951d4d75e584811be86203d11fc63aa90e229197376de2f9cb
Jul 28 17:18:00.313238 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.313209    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cqcfv"]
Jul 28 17:18:00.315920 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:00.315885    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef621fc_443a_4f17_8c74_bbe3605a4c89.slice/crio-ae4164c7214dd1dd7b5bc0488bcd9759dbd4ace39a3623a9e5626e5cdde3186e WatchSource:0}: Error finding container ae4164c7214dd1dd7b5bc0488bcd9759dbd4ace39a3623a9e5626e5cdde3186e: Status 404 returned error can't find the container with id ae4164c7214dd1dd7b5bc0488bcd9759dbd4ace39a3623a9e5626e5cdde3186e
Jul 28 17:18:00.629608 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.629546    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gsqc" event={"ID":"1d9f58fc-5928-4213-8e45-380962645bd1","Type":"ContainerStarted","Data":"5dfa3cd1c86435951d4d75e584811be86203d11fc63aa90e229197376de2f9cb"}
Jul 28 17:18:00.630601 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.630572    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-27fcp" event={"ID":"8f4bfba2-733c-453a-a03d-02d4994797d4","Type":"ContainerStarted","Data":"3df3d1f1e739e89f7034887317675795a84edc7876e87921ab3450e4766392b3"}
Jul 28 17:18:00.631501 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:00.631480    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cqcfv" event={"ID":"def621fc-443a-4f17-8c74-bbe3605a4c89","Type":"ContainerStarted","Data":"ae4164c7214dd1dd7b5bc0488bcd9759dbd4ace39a3623a9e5626e5cdde3186e"}
Jul 28 17:18:05.044742 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.044704    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:18:05.045147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.044782    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:18:05.045677 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:18:05.045652    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:05.045787 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:18:05.045716    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls podName:7bb9c484-525b-430e-bee3-69208d19d269 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:21.045696848 +0000 UTC m=+86.389289120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bttvz" (UID: "7bb9c484-525b-430e-bee3-69208d19d269") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:05.045920 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.045907    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bb9c484-525b-430e-bee3-69208d19d269-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:18:05.652783 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.652731    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cqcfv" event={"ID":"def621fc-443a-4f17-8c74-bbe3605a4c89","Type":"ContainerStarted","Data":"53a0a715e93c0b482405894033a7266d464ed9c4db2dc9ede51d0c40e1c8a705"}
Jul 28 17:18:05.652992 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.652800    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cqcfv" event={"ID":"def621fc-443a-4f17-8c74-bbe3605a4c89","Type":"ContainerStarted","Data":"a7c3d335f8e00417dcb2cc92a44672589c27154db2b625cc4433ab91ca972024"}
Jul 28 17:18:05.652992 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.652908    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:18:05.654137 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.654111    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gsqc" event={"ID":"1d9f58fc-5928-4213-8e45-380962645bd1","Type":"ContainerStarted","Data":"8747ef4b3a831b8c59db4f5a89352e4f211e8bbbf2f949ad9409861e47c4219c"}
Jul 28 17:18:05.654236 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.654165    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:18:05.655459 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.655439    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-27fcp" event={"ID":"8f4bfba2-733c-453a-a03d-02d4994797d4","Type":"ContainerStarted","Data":"9c4fe991af8900ee8259fbdfa44d53a48e63ff8bb971667cd4c180ed5ac54113"}
Jul 28 17:18:05.676874 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.676821    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cqcfv" podStartSLOduration=33.05557892 podStartE2EDuration="37.676809117s" totalImagesPullingTime="4.621230197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:05.676027246 +0000 UTC m=+71.019619522" watchObservedRunningTime="2026-07-28 17:18:05.676809117 +0000 UTC m=+71.020401428"
Jul 28 17:18:05.693709 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.693670    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-27fcp" podStartSLOduration=65.977130962 podStartE2EDuration="1m10.693657446s" totalImagesPullingTime="4.716526484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:05.69290411 +0000 UTC m=+71.036496389" watchObservedRunningTime="2026-07-28 17:18:05.693657446 +0000 UTC m=+71.037249722"
Jul 28 17:18:05.708706 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:05.708668    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4gsqc" podStartSLOduration=66.014934441 podStartE2EDuration="1m10.708660453s" totalImagesPullingTime="4.693726012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:05.708333246 +0000 UTC m=+71.051925525" watchObservedRunningTime="2026-07-28 17:18:05.708660453 +0000 UTC m=+71.052252760"
Jul 28 17:18:13.836239 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.836206    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"]
Jul 28 17:18:13.838298 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.838281    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:13.843024 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.842997    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 17:18:13.843181 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.843067    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 17:18:13.843181 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.843080    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:18:13.843181 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.843119    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:18:13.843419 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.843351    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 17:18:13.843591 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.843575    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:18:13.843672 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.843606    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 17:18:13.863451 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.863431    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"]
Jul 28 17:18:13.911083 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.911054    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-hub\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:13.911215 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.911086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:13.911215 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.911116    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:13.911215 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.911167    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mng\" (UniqueName: \"kubernetes.io/projected/e6713d1a-930b-42b1-82b1-17f571a011d7-kube-api-access-b6mng\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:13.911329 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.911225    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-ca\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:13.911329 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:13.911254    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6713d1a-930b-42b1-82b1-17f571a011d7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.011968 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.011930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-hub\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.011968 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.011968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.012213 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.011996    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.012213 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.012019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mng\" (UniqueName: \"kubernetes.io/projected/e6713d1a-930b-42b1-82b1-17f571a011d7-kube-api-access-b6mng\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.012213 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.012070    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-ca\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.012213 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.012107    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6713d1a-930b-42b1-82b1-17f571a011d7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.012965 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.012932    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6713d1a-930b-42b1-82b1-17f571a011d7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.014761 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.014736    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.014761 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.014744    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.014888 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.014746    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-ca\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.014927 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.014912    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e6713d1a-930b-42b1-82b1-17f571a011d7-hub\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.031480 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.031451    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mng\" (UniqueName: \"kubernetes.io/projected/e6713d1a-930b-42b1-82b1-17f571a011d7-kube-api-access-b6mng\") pod \"cluster-proxy-proxy-agent-67cbc84b88-cwb7q\" (UID: \"e6713d1a-930b-42b1-82b1-17f571a011d7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.158699 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.158619    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"
Jul 28 17:18:14.284876 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.284836    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q"]
Jul 28 17:18:14.287898 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:14.287877    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6713d1a_930b_42b1_82b1_17f571a011d7.slice/crio-7bea919471ad59d414d98c423f5367917b03ceddb39d7001a0b2d478b2ccf6a5 WatchSource:0}: Error finding container 7bea919471ad59d414d98c423f5367917b03ceddb39d7001a0b2d478b2ccf6a5: Status 404 returned error can't find the container with id 7bea919471ad59d414d98c423f5367917b03ceddb39d7001a0b2d478b2ccf6a5
Jul 28 17:18:14.682787 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:14.682757    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q" event={"ID":"e6713d1a-930b-42b1-82b1-17f571a011d7","Type":"ContainerStarted","Data":"7bea919471ad59d414d98c423f5367917b03ceddb39d7001a0b2d478b2ccf6a5"}
Jul 28 17:18:15.661114 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:15.661080    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cqcfv"
Jul 28 17:18:15.727883 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:15.726239    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:15.727883 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:15.726385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:15.727883 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:15.727114    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cc057ee-b81c-4977-ba8a-b8a2f478729b-service-ca-bundle\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:15.730587 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:15.730541    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc057ee-b81c-4977-ba8a-b8a2f478729b-metrics-certs\") pod \"router-default-66dc97f78c-hsl5f\" (UID: \"3cc057ee-b81c-4977-ba8a-b8a2f478729b\") " pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:15.970848 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:15.970766    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xj5xx\""
Jul 28 17:18:15.979134 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:15.979096    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:16.474136 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:16.474103    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66dc97f78c-hsl5f"]
Jul 28 17:18:16.476384 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:16.476359    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc057ee_b81c_4977_ba8a_b8a2f478729b.slice/crio-ffe4892e0e5ed8a5d36e23e148452c89249fbf5f18fcb7b7533ffc53bc97fbab WatchSource:0}: Error finding container ffe4892e0e5ed8a5d36e23e148452c89249fbf5f18fcb7b7533ffc53bc97fbab: Status 404 returned error can't find the container with id ffe4892e0e5ed8a5d36e23e148452c89249fbf5f18fcb7b7533ffc53bc97fbab
Jul 28 17:18:16.692384 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:16.692289    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66dc97f78c-hsl5f" event={"ID":"3cc057ee-b81c-4977-ba8a-b8a2f478729b","Type":"ContainerStarted","Data":"32ced0fe4c40b6054e97d6346b930dfd8e662927c9d4398fbc0f6c9d7bef1711"}
Jul 28 17:18:16.692384 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:16.692350    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66dc97f78c-hsl5f" event={"ID":"3cc057ee-b81c-4977-ba8a-b8a2f478729b","Type":"ContainerStarted","Data":"ffe4892e0e5ed8a5d36e23e148452c89249fbf5f18fcb7b7533ffc53bc97fbab"}
Jul 28 17:18:16.713590 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:16.713497    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66dc97f78c-hsl5f" podStartSLOduration=33.713475035 podStartE2EDuration="33.713475035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:16.712164383 +0000 UTC m=+82.055756689" watchObservedRunningTime="2026-07-28 17:18:16.713475035 +0000 UTC m=+82.057067313"
Jul 28 17:18:16.980270 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:16.980102    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:16.982774 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:16.982745    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:17.695949 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:17.695925    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:17.697122 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:17.697101    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66dc97f78c-hsl5f"
Jul 28 17:18:18.704353 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:18.704315    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q" event={"ID":"e6713d1a-930b-42b1-82b1-17f571a011d7","Type":"ContainerStarted","Data":"c9a2333b2e1d771029d4cf97a55d8daf5876bf7cd24e04720dd289a20429eecc"}
Jul 28 17:18:18.704353 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:18.704359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q" event={"ID":"e6713d1a-930b-42b1-82b1-17f571a011d7","Type":"ContainerStarted","Data":"e0bd5bbe82a0b7a184863600b756174c4ac3b4326f7688336025bc9f12c4b449"}
Jul 28 17:18:18.704894 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:18.704373    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q" event={"ID":"e6713d1a-930b-42b1-82b1-17f571a011d7","Type":"ContainerStarted","Data":"842b079d6827e3beff456ca543106ca6ec9909cfba09309bacfc297ab35a408e"}
Jul 28 17:18:18.725196 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:18.725150    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67cbc84b88-cwb7q" podStartSLOduration=2.31790076 podStartE2EDuration="5.725133254s" totalImagesPullingTime="3.407232494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:18.72367973 +0000 UTC m=+84.067272009" watchObservedRunningTime="2026-07-28 17:18:18.725133254 +0000 UTC m=+84.068725539"
Jul 28 17:18:21.068776 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:21.068737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:18:21.071028 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:21.071004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bb9c484-525b-430e-bee3-69208d19d269-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bttvz\" (UID: \"7bb9c484-525b-430e-bee3-69208d19d269\") " pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:18:21.110400 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:21.110379    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hbmbv\""
Jul 28 17:18:21.118652 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:21.118635    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bttvz"
Jul 28 17:18:21.243668 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:21.243641    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bttvz"]
Jul 28 17:18:21.246351 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:21.246324    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb9c484_525b_430e_bee3_69208d19d269.slice/crio-56ce5a1b849b14129a304d243553068168135e15aa8b022899fda9a97fbef48b WatchSource:0}: Error finding container 56ce5a1b849b14129a304d243553068168135e15aa8b022899fda9a97fbef48b: Status 404 returned error can't find the container with id 56ce5a1b849b14129a304d243553068168135e15aa8b022899fda9a97fbef48b
Jul 28 17:18:21.716754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:21.716676    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bttvz" event={"ID":"7bb9c484-525b-430e-bee3-69208d19d269","Type":"ContainerStarted","Data":"56ce5a1b849b14129a304d243553068168135e15aa8b022899fda9a97fbef48b"}
Jul 28 17:18:26.741368 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:26.741329    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bttvz" event={"ID":"7bb9c484-525b-430e-bee3-69208d19d269","Type":"ContainerStarted","Data":"c3e0b99fc393f47845098cd14b62339be93b8a92a19e4fd0d75d006abffa67a0"}
Jul 28 17:18:29.754318 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:29.754276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bttvz" event={"ID":"7bb9c484-525b-430e-bee3-69208d19d269","Type":"ContainerStarted","Data":"cac6098edcf1595ba7932dcc96986b435a2c4e71acc69cf48fef0d9f721b5557"}
Jul 28 17:18:31.586517 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.586478    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"]
Jul 28 17:18:31.591313 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.591290    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.594068 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.594044    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jgxm7\""
Jul 28 17:18:31.594181 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.594144    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:18:31.594311 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.594202    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 17:18:31.595354 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.595332    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:18:31.595461 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.595368    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:18:31.595525 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.595481    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:18:31.602941 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.602916    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"]
Jul 28 17:18:31.628173 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.628151    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n8mhj"]
Jul 28 17:18:31.631374 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.631353    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.635728 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.635701    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:18:31.635868 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.635797    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:18:31.635868 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.635824    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kzkgt\""
Jul 28 17:18:31.635868 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.635851    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:18:31.656855 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.656832    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9w9w\" (UniqueName: \"kubernetes.io/projected/99103575-d3da-4012-b08a-ec2204751a9f-kube-api-access-d9w9w\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.656988 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.656959    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.657055 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.657000    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99103575-d3da-4012-b08a-ec2204751a9f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.657113 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.657067    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.757717 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.757678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.757900 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.757725    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-wtmp\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.757900 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.757798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9w9w\" (UniqueName: \"kubernetes.io/projected/99103575-d3da-4012-b08a-ec2204751a9f-kube-api-access-d9w9w\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.757900 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.757863    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-textfile\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.757900 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.757889    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-root\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.758131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.757951    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zpc\" (UniqueName: \"kubernetes.io/projected/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-kube-api-access-l6zpc\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.758131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.758022    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-metrics-client-ca\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.758131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.758059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-sys\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.758131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.758097    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.758131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.758124    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-tls\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.758376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.758154    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.758376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.758235    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99103575-d3da-4012-b08a-ec2204751a9f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.758376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.758278    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.758562 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:18:31.758384    2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 17:18:31.758562 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:18:31.758455    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-tls podName:99103575-d3da-4012-b08a-ec2204751a9f nodeName:}" failed. No retries permitted until 2026-07-28 17:18:32.258431824 +0000 UTC m=+97.602024079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-4fth6" (UID: "99103575-d3da-4012-b08a-ec2204751a9f") : secret "openshift-state-metrics-tls" not found
Jul 28 17:18:31.759089 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.759065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99103575-d3da-4012-b08a-ec2204751a9f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.761027 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.761004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.774123 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.774101    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9w9w\" (UniqueName: \"kubernetes.io/projected/99103575-d3da-4012-b08a-ec2204751a9f-kube-api-access-d9w9w\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:31.859575 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859480    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-sys\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.859746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.859746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859581    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-sys\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.859746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859605    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-tls\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.859746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859665    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.859746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859696    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-wtmp\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.859746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-textfile\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860110 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859767    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-root\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860110 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zpc\" (UniqueName: \"kubernetes.io/projected/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-kube-api-access-l6zpc\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860110 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.859845    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-metrics-client-ca\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860110 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.860073    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-root\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860300 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.860180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-wtmp\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860300 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.860227    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-textfile\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860464 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.860440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-metrics-client-ca\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.860605 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.860534    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.862069 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.862048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-tls\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.862221 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.862203    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.878871 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.878848    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zpc\" (UniqueName: \"kubernetes.io/projected/a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3-kube-api-access-l6zpc\") pod \"node-exporter-n8mhj\" (UID: \"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3\") " pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:31.942018 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:31.941987    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n8mhj"
Jul 28 17:18:32.263224 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:32.263150    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:32.265603 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:32.265579    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99103575-d3da-4012-b08a-ec2204751a9f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4fth6\" (UID: \"99103575-d3da-4012-b08a-ec2204751a9f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:32.501750 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:32.501716    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"
Jul 28 17:18:33.173474 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:33.173434    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b3c2e6_55fc_4c66_ab16_c5a46ae2bbd3.slice/crio-03e083476ad6f5b7014f19dc622043a298474be7eb7ca9d0735674f8426b9bfd WatchSource:0}: Error finding container 03e083476ad6f5b7014f19dc622043a298474be7eb7ca9d0735674f8426b9bfd: Status 404 returned error can't find the container with id 03e083476ad6f5b7014f19dc622043a298474be7eb7ca9d0735674f8426b9bfd
Jul 28 17:18:33.286983 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:33.286950    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6"]
Jul 28 17:18:33.290486 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:18:33.290453    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99103575_d3da_4012_b08a_ec2204751a9f.slice/crio-3b12adcf8453b3dd2c3052c5f8ea3677b1ae2e93cf82b8545e35f67411ac618a WatchSource:0}: Error finding container 3b12adcf8453b3dd2c3052c5f8ea3677b1ae2e93cf82b8545e35f67411ac618a: Status 404 returned error can't find the container with id 3b12adcf8453b3dd2c3052c5f8ea3677b1ae2e93cf82b8545e35f67411ac618a
Jul 28 17:18:33.770706 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:33.770653    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bttvz" event={"ID":"7bb9c484-525b-430e-bee3-69208d19d269","Type":"ContainerStarted","Data":"cc60822fff69633ea0345743614915df8a2288fc8c112173e6aa93527613da58"}
Jul 28 17:18:33.772881 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:33.772846    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6" event={"ID":"99103575-d3da-4012-b08a-ec2204751a9f","Type":"ContainerStarted","Data":"9304990dd582c0e7b6f5960a52d6497f329d6727173ed38bde5baa2fb30c8cc6"}
Jul 28 17:18:33.773020 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:33.772906    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6" event={"ID":"99103575-d3da-4012-b08a-ec2204751a9f","Type":"ContainerStarted","Data":"da3f570f5a25e2a3f5e8f9b97f498bc1341f2f5e5800cd16faa5496b0ece6a2a"}
Jul 28 17:18:33.773020 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:33.772941    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6" event={"ID":"99103575-d3da-4012-b08a-ec2204751a9f","Type":"ContainerStarted","Data":"3b12adcf8453b3dd2c3052c5f8ea3677b1ae2e93cf82b8545e35f67411ac618a"}
Jul 28 17:18:33.774056 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:33.774032    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8mhj" event={"ID":"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3","Type":"ContainerStarted","Data":"03e083476ad6f5b7014f19dc622043a298474be7eb7ca9d0735674f8426b9bfd"}
Jul 28 17:18:33.799152 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:33.799114    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bttvz" podStartSLOduration=32.936295714 podStartE2EDuration="44.799100445s" totalImagesPullingTime="11.862804731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:33.796688348 +0000 UTC m=+99.140280624" watchObservedRunningTime="2026-07-28 17:18:33.799100445 +0000 UTC m=+99.142692721"
Jul 28 17:18:34.778961 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:34.778929    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6" event={"ID":"99103575-d3da-4012-b08a-ec2204751a9f","Type":"ContainerStarted","Data":"e2f81d57c42706114d818de05b749f59d87ee155dbe2621f2fefe5a64b8b14d8"}
Jul 28 17:18:34.780376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:34.780354    2585 generic.go:356] "Generic (PLEG): container finished" podID="a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3" containerID="ecd308b87b2a7c83609d5cd1632d32260dabaeacd81ed1b3f07e1cefcf51333c" exitCode=0
Jul 28 17:18:34.780483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:34.780438    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8mhj" event={"ID":"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3","Type":"ContainerDied","Data":"ecd308b87b2a7c83609d5cd1632d32260dabaeacd81ed1b3f07e1cefcf51333c"}
Jul 28 17:18:34.800174 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:34.800132    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4fth6" podStartSLOduration=2.801974651 podStartE2EDuration="3.800118834s" totalImagesPullingTime="998.144183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:34.799808689 +0000 UTC m=+100.143400970" watchObservedRunningTime="2026-07-28 17:18:34.800118834 +0000 UTC m=+100.143711111"
Jul 28 17:18:35.785842 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:35.785798    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8mhj" event={"ID":"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3","Type":"ContainerStarted","Data":"dff36ef201c71b87bfcd06d48c4324001c059ff3b04b657dca585daf6be0493a"}
Jul 28 17:18:35.786217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:35.785865    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8mhj" event={"ID":"a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3","Type":"ContainerStarted","Data":"ec6f998bbb03cd369fb0c8e4ff88191ac5194b08557ec20311ea1ca64356a1ad"}
Jul 28 17:18:35.816929 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:35.816887    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n8mhj" podStartSLOduration=3.512901066 podStartE2EDuration="4.816873175s" totalImagesPullingTime="1.236645357s" totalInitContainerRuntime="67.326752ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:35.815177921 +0000 UTC m=+101.158770218" watchObservedRunningTime="2026-07-28 17:18:35.816873175 +0000 UTC m=+101.160465453"
Jul 28 17:18:36.661813 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:18:36.661782    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4gsqc"
Jul 28 17:19:03.880313 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:03.880237    2585 generic.go:356] "Generic (PLEG): container finished" podID="76fd56c1-4eba-4445-a7b5-a75f304f150b" containerID="9f2efb1da22f1af8edce6dffd505d3bca67d559d46592056c9e57e4b2166d627" exitCode=0
Jul 28 17:19:03.880717 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:03.880310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf" event={"ID":"76fd56c1-4eba-4445-a7b5-a75f304f150b","Type":"ContainerDied","Data":"9f2efb1da22f1af8edce6dffd505d3bca67d559d46592056c9e57e4b2166d627"}
Jul 28 17:19:03.880717 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:03.880635    2585 scope.go:117] "RemoveContainer" containerID="9f2efb1da22f1af8edce6dffd505d3bca67d559d46592056c9e57e4b2166d627"
Jul 28 17:19:04.003210 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:04.003174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:19:04.005483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:04.005461    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ff851-6c38-46db-b020-57ae09a57aa3-metrics-certs\") pod \"network-metrics-daemon-ng94v\" (UID: \"0a3ff851-6c38-46db-b020-57ae09a57aa3\") " pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:19:04.187723 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:04.187645    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2k2q\""
Jul 28 17:19:04.196066 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:04.196047    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ng94v"
Jul 28 17:19:04.364058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:04.364029    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ng94v"]
Jul 28 17:19:04.367367 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:19:04.367341    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ff851_6c38_46db_b020_57ae09a57aa3.slice/crio-b3433498f013019061b46b5a95b994543960eda8d5538f5d762019d5054be17c WatchSource:0}: Error finding container b3433498f013019061b46b5a95b994543960eda8d5538f5d762019d5054be17c: Status 404 returned error can't find the container with id b3433498f013019061b46b5a95b994543960eda8d5538f5d762019d5054be17c
Jul 28 17:19:04.884540 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:04.884501    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ng94v" event={"ID":"0a3ff851-6c38-46db-b020-57ae09a57aa3","Type":"ContainerStarted","Data":"b3433498f013019061b46b5a95b994543960eda8d5538f5d762019d5054be17c"}
Jul 28 17:19:04.886328 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:04.886303    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf" event={"ID":"76fd56c1-4eba-4445-a7b5-a75f304f150b","Type":"ContainerStarted","Data":"d46b68234697188b0087aebf5507c334ae01a523e9c942de476f659587bc1be7"}
Jul 28 17:19:05.891020 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:05.890984    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ng94v" event={"ID":"0a3ff851-6c38-46db-b020-57ae09a57aa3","Type":"ContainerStarted","Data":"b91df852f6cf80c30501232e0730a46538a99c4155c2831d50eae3d357da0dda"}
Jul 28 17:19:05.891446 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:05.891036    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ng94v" event={"ID":"0a3ff851-6c38-46db-b020-57ae09a57aa3","Type":"ContainerStarted","Data":"c5937d874af62a5123bb8220496433dc87d8b6a6d53441235c212f81c8ef8d6e"}
Jul 28 17:19:05.914723 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:05.914681    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ng94v" podStartSLOduration=129.909515664 podStartE2EDuration="2m10.914666405s" totalImagesPullingTime="1.005150741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:05.913689407 +0000 UTC m=+131.257281683" watchObservedRunningTime="2026-07-28 17:19:05.914666405 +0000 UTC m=+131.258258681"
Jul 28 17:19:08.904191 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:08.904156    2585 generic.go:356] "Generic (PLEG): container finished" podID="28a356b3-2909-4397-8373-0eaf6ba7a5e5" containerID="b4e563d054ed158434d1d45c310badfd2afb55e3c58feef12631c92995058852" exitCode=0
Jul 28 17:19:08.904652 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:08.904204    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd" event={"ID":"28a356b3-2909-4397-8373-0eaf6ba7a5e5","Type":"ContainerDied","Data":"b4e563d054ed158434d1d45c310badfd2afb55e3c58feef12631c92995058852"}
Jul 28 17:19:08.904652 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:08.904534    2585 scope.go:117] "RemoveContainer" containerID="b4e563d054ed158434d1d45c310badfd2afb55e3c58feef12631c92995058852"
Jul 28 17:19:09.910433 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:19:09.910390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-d65fd" event={"ID":"28a356b3-2909-4397-8373-0eaf6ba7a5e5","Type":"ContainerStarted","Data":"87ac43ce9ed5de7557f6a69fad9826434b0f01c512b9cbc3cc298e7f0bb8620b"}
Jul 28 17:21:43.873319 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.873282    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-l9spq"]
Jul 28 17:21:43.876304 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.876288    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:43.879132 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.879109    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:21:43.879273 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.879176    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:21:43.879400 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.879386    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 17:21:43.880370 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.880351    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 17:21:43.880448 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.880431    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6lvgw\""
Jul 28 17:21:43.886981 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:43.886958    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-l9spq"]
Jul 28 17:21:44.019218 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.019179    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73575839-a534-413c-8165-6b9e9da6dade-certificates\") pod \"keda-admission-56f9798bb4-l9spq\" (UID: \"73575839-a534-413c-8165-6b9e9da6dade\") " pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:44.019391 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.019227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlph\" (UniqueName: \"kubernetes.io/projected/73575839-a534-413c-8165-6b9e9da6dade-kube-api-access-mzlph\") pod \"keda-admission-56f9798bb4-l9spq\" (UID: \"73575839-a534-413c-8165-6b9e9da6dade\") " pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:44.120054 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.120023    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73575839-a534-413c-8165-6b9e9da6dade-certificates\") pod \"keda-admission-56f9798bb4-l9spq\" (UID: \"73575839-a534-413c-8165-6b9e9da6dade\") " pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:44.120230 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.120064    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlph\" (UniqueName: \"kubernetes.io/projected/73575839-a534-413c-8165-6b9e9da6dade-kube-api-access-mzlph\") pod \"keda-admission-56f9798bb4-l9spq\" (UID: \"73575839-a534-413c-8165-6b9e9da6dade\") " pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:44.122516 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.122483    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73575839-a534-413c-8165-6b9e9da6dade-certificates\") pod \"keda-admission-56f9798bb4-l9spq\" (UID: \"73575839-a534-413c-8165-6b9e9da6dade\") " pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:44.130954 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.130897    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlph\" (UniqueName: \"kubernetes.io/projected/73575839-a534-413c-8165-6b9e9da6dade-kube-api-access-mzlph\") pod \"keda-admission-56f9798bb4-l9spq\" (UID: \"73575839-a534-413c-8165-6b9e9da6dade\") " pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:44.185865 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.185839    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:44.313379 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.313350    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-l9spq"]
Jul 28 17:21:44.316527 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:21:44.316503    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73575839_a534_413c_8165_6b9e9da6dade.slice/crio-218851d9129e92db070152fc722028fae670cb7d013afb26091461d95add489b WatchSource:0}: Error finding container 218851d9129e92db070152fc722028fae670cb7d013afb26091461d95add489b: Status 404 returned error can't find the container with id 218851d9129e92db070152fc722028fae670cb7d013afb26091461d95add489b
Jul 28 17:21:44.432109 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:44.432020    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-l9spq" event={"ID":"73575839-a534-413c-8165-6b9e9da6dade","Type":"ContainerStarted","Data":"218851d9129e92db070152fc722028fae670cb7d013afb26091461d95add489b"}
Jul 28 17:21:47.444092 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:47.444054    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-l9spq" event={"ID":"73575839-a534-413c-8165-6b9e9da6dade","Type":"ContainerStarted","Data":"9cd62108190957d9509775445d939140fb16732cf4cb7b13c640897566567534"}
Jul 28 17:21:47.444586 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:47.444179    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:21:47.464527 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:47.464478    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-l9spq" podStartSLOduration=2.171331597 podStartE2EDuration="4.46446461s" totalImagesPullingTime="2.293133013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:47.462887436 +0000 UTC m=+292.806479713" watchObservedRunningTime="2026-07-28 17:21:47.46446461 +0000 UTC m=+292.808057116"
Jul 28 17:21:55.010220 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:55.010193    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:21:55.014760 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:55.014741    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:21:55.015825 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:21:55.015809    2585 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:22:08.450140 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:22:08.450107    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-l9spq"
Jul 28 17:23:02.210053 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.209967    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"]
Jul 28 17:23:02.213160 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.213139    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.215851 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.215824    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:23:02.217285 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.217266    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:23:02.217396 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.217273    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 17:23:02.217396 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.217299    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-llgjh\""
Jul 28 17:23:02.222708 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.222687    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"]
Jul 28 17:23:02.250880 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.250853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/273fcc51-0370-4f95-82d2-a49bbe29db93-cert\") pod \"kserve-controller-manager-85d8b8ddfc-dv2cm\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.251021 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.250911    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58kwm\" (UniqueName: \"kubernetes.io/projected/273fcc51-0370-4f95-82d2-a49bbe29db93-kube-api-access-58kwm\") pod \"kserve-controller-manager-85d8b8ddfc-dv2cm\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.351507 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.351466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/273fcc51-0370-4f95-82d2-a49bbe29db93-cert\") pod \"kserve-controller-manager-85d8b8ddfc-dv2cm\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.351692 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.351519    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58kwm\" (UniqueName: \"kubernetes.io/projected/273fcc51-0370-4f95-82d2-a49bbe29db93-kube-api-access-58kwm\") pod \"kserve-controller-manager-85d8b8ddfc-dv2cm\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.354000 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.353978    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/273fcc51-0370-4f95-82d2-a49bbe29db93-cert\") pod \"kserve-controller-manager-85d8b8ddfc-dv2cm\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.363137 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.363111    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58kwm\" (UniqueName: \"kubernetes.io/projected/273fcc51-0370-4f95-82d2-a49bbe29db93-kube-api-access-58kwm\") pod \"kserve-controller-manager-85d8b8ddfc-dv2cm\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.523395 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.523368    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:02.650781 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.650749    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"]
Jul 28 17:23:02.653668 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:23:02.653642    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273fcc51_0370_4f95_82d2_a49bbe29db93.slice/crio-3b38a2a46802a9e7178cd78d517d2220e27876cc42403081e4b2365ddc3d57d9 WatchSource:0}: Error finding container 3b38a2a46802a9e7178cd78d517d2220e27876cc42403081e4b2365ddc3d57d9: Status 404 returned error can't find the container with id 3b38a2a46802a9e7178cd78d517d2220e27876cc42403081e4b2365ddc3d57d9
Jul 28 17:23:02.654880 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.654861    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:23:02.702337 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:02.702300    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm" event={"ID":"273fcc51-0370-4f95-82d2-a49bbe29db93","Type":"ContainerStarted","Data":"3b38a2a46802a9e7178cd78d517d2220e27876cc42403081e4b2365ddc3d57d9"}
Jul 28 17:23:05.716054 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:05.716013    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm" event={"ID":"273fcc51-0370-4f95-82d2-a49bbe29db93","Type":"ContainerStarted","Data":"4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8"}
Jul 28 17:23:05.716471 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:05.716159    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:23:05.733804 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:05.733759    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm" podStartSLOduration=1.375271535 podStartE2EDuration="3.733744911s" totalImagesPullingTime="2.358473376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:05.733287651 +0000 UTC m=+371.076879927" watchObservedRunningTime="2026-07-28 17:23:05.733744911 +0000 UTC m=+371.077337241"
Jul 28 17:23:36.724465 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:23:36.724430    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:24:13.506697 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.506653    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ccdhq"]
Jul 28 17:24:13.510000 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.509976    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.513586 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.513542    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 17:24:13.514730 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.514709    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-99mf4\""
Jul 28 17:24:13.514836 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.514745    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 17:24:13.520417 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.520391    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ccdhq"]
Jul 28 17:24:13.680531 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.680494    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7fdece9-07e3-49f4-981c-d2886e3b2565-bound-sa-token\") pod \"cert-manager-55b4d954d5-ccdhq\" (UID: \"b7fdece9-07e3-49f4-981c-d2886e3b2565\") " pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.680732 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.680567    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7xs5\" (UniqueName: \"kubernetes.io/projected/b7fdece9-07e3-49f4-981c-d2886e3b2565-kube-api-access-s7xs5\") pod \"cert-manager-55b4d954d5-ccdhq\" (UID: \"b7fdece9-07e3-49f4-981c-d2886e3b2565\") " pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.781380 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.781348    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7fdece9-07e3-49f4-981c-d2886e3b2565-bound-sa-token\") pod \"cert-manager-55b4d954d5-ccdhq\" (UID: \"b7fdece9-07e3-49f4-981c-d2886e3b2565\") " pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.781545 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.781396    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7xs5\" (UniqueName: \"kubernetes.io/projected/b7fdece9-07e3-49f4-981c-d2886e3b2565-kube-api-access-s7xs5\") pod \"cert-manager-55b4d954d5-ccdhq\" (UID: \"b7fdece9-07e3-49f4-981c-d2886e3b2565\") " pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.790430 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.790401    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7xs5\" (UniqueName: \"kubernetes.io/projected/b7fdece9-07e3-49f4-981c-d2886e3b2565-kube-api-access-s7xs5\") pod \"cert-manager-55b4d954d5-ccdhq\" (UID: \"b7fdece9-07e3-49f4-981c-d2886e3b2565\") " pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.790530 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.790441    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7fdece9-07e3-49f4-981c-d2886e3b2565-bound-sa-token\") pod \"cert-manager-55b4d954d5-ccdhq\" (UID: \"b7fdece9-07e3-49f4-981c-d2886e3b2565\") " pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.819468 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.819434    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ccdhq"
Jul 28 17:24:13.943992 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.943968    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ccdhq"]
Jul 28 17:24:13.946186 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:24:13.946162    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7fdece9_07e3_49f4_981c_d2886e3b2565.slice/crio-d58d43eb1d67b1787c606749191ef4ce9423953fd2ddec96a07ae2e6223f6e56 WatchSource:0}: Error finding container d58d43eb1d67b1787c606749191ef4ce9423953fd2ddec96a07ae2e6223f6e56: Status 404 returned error can't find the container with id d58d43eb1d67b1787c606749191ef4ce9423953fd2ddec96a07ae2e6223f6e56
Jul 28 17:24:13.952305 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:13.952278    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ccdhq" event={"ID":"b7fdece9-07e3-49f4-981c-d2886e3b2565","Type":"ContainerStarted","Data":"d58d43eb1d67b1787c606749191ef4ce9423953fd2ddec96a07ae2e6223f6e56"}
Jul 28 17:24:17.969601 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:17.969502    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ccdhq" event={"ID":"b7fdece9-07e3-49f4-981c-d2886e3b2565","Type":"ContainerStarted","Data":"9c09d8109b261d02827fe484386df7f5d3a495854063ad81d3b28391446e22db"}
Jul 28 17:24:17.989359 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:17.989310    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ccdhq" podStartSLOduration=1.41607679 podStartE2EDuration="4.989295842s" totalImagesPullingTime="3.573219052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:17.987989178 +0000 UTC m=+443.331581455" watchObservedRunningTime="2026-07-28 17:24:17.989295842 +0000 UTC m=+443.332888119"
Jul 28 17:24:42.367790 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.367753    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"]
Jul 28 17:24:42.370826 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.370799    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.373832 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.373804    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-cp9nf\""
Jul 28 17:24:42.373977 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.373835    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 17:24:42.383029 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.383001    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"]
Jul 28 17:24:42.393883 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.393851    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.393990 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.393901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.393990 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.393931    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.393990 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.393970    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.394084 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.393995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.394084 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.394037    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.394147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.394100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.394147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.394132    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.394206 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.394188    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566xf\" (UniqueName: \"kubernetes.io/projected/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-kube-api-access-566xf\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494707 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566xf\" (UniqueName: \"kubernetes.io/projected/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-kube-api-access-566xf\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494716    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494743    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494766    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494801    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494823    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494865    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.494918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.494918    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.495284 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.495202    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.495344 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.495282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.495398 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.495348    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.495398 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.495362    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.495775 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.495752    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.497145 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.497121    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.497375 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.497354    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.512282 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.512223    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.512664 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.512640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-566xf\" (UniqueName: \"kubernetes.io/projected/c4c71c04-b83e-44b3-b3ce-4a06dfec9307-kube-api-access-566xf\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l\" (UID: \"c4c71c04-b83e-44b3-b3ce-4a06dfec9307\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.681572 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.681457    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:42.813124 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:42.813101    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"]
Jul 28 17:24:42.815293 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:24:42.815269    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c71c04_b83e_44b3_b3ce_4a06dfec9307.slice/crio-e6bbf684767093a03e4a545a7705f1371288972450a694495dab45e63fd086b5 WatchSource:0}: Error finding container e6bbf684767093a03e4a545a7705f1371288972450a694495dab45e63fd086b5: Status 404 returned error can't find the container with id e6bbf684767093a03e4a545a7705f1371288972450a694495dab45e63fd086b5
Jul 28 17:24:43.061683 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:43.061648    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" event={"ID":"c4c71c04-b83e-44b3-b3ce-4a06dfec9307","Type":"ContainerStarted","Data":"e6bbf684767093a03e4a545a7705f1371288972450a694495dab45e63fd086b5"}
Jul 28 17:24:45.780248 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:45.780210    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 17:24:45.780545 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:45.780287    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 17:24:45.780545 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:45.780318    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 17:24:46.077046 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:46.077005    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" event={"ID":"c4c71c04-b83e-44b3-b3ce-4a06dfec9307","Type":"ContainerStarted","Data":"82fe3ae95ffcbea11d7831a576d0a1f6c1009b3f1c05320e3c452d7641801144"}
Jul 28 17:24:46.099108 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:46.099055    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" podStartSLOduration=1.136255128 podStartE2EDuration="4.099040043s" totalImagesPullingTime="2.962784915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:46.099015059 +0000 UTC m=+471.442607337" watchObservedRunningTime="2026-07-28 17:24:46.099040043 +0000 UTC m=+471.442632320"
Jul 28 17:24:46.681974 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:46.681936    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:46.686355 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:46.686331    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:47.081107 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:47.081077    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:47.082205 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:47.082188    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:24:53.757208 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.757176    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"]
Jul 28 17:24:53.761764 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.761744    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.765745 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.765722    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 17:24:53.765871 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.765806    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 17:24:53.765871 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.765830    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 17:24:53.765871 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.765858    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:24:53.766035 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.765878    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pxkbp\""
Jul 28 17:24:53.766035 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.765915    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 17:24:53.776130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.776108    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"]
Jul 28 17:24:53.891264 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.891230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlsp8\" (UniqueName: \"kubernetes.io/projected/f461e503-ea75-4f23-ae41-a85adacc2826-kube-api-access-rlsp8\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.891467 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.891281    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f461e503-ea75-4f23-ae41-a85adacc2826-manager-config\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.891467 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.891372    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f461e503-ea75-4f23-ae41-a85adacc2826-cert\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.891467 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.891416    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f461e503-ea75-4f23-ae41-a85adacc2826-metrics-cert\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.992610 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.992567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f461e503-ea75-4f23-ae41-a85adacc2826-manager-config\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.992796 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.992649    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f461e503-ea75-4f23-ae41-a85adacc2826-cert\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.992796 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.992680    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f461e503-ea75-4f23-ae41-a85adacc2826-metrics-cert\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.992796 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.992710    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlsp8\" (UniqueName: \"kubernetes.io/projected/f461e503-ea75-4f23-ae41-a85adacc2826-kube-api-access-rlsp8\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.993337 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.993314    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f461e503-ea75-4f23-ae41-a85adacc2826-manager-config\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.995164 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.995145    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f461e503-ea75-4f23-ae41-a85adacc2826-cert\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:53.995241 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:53.995223    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f461e503-ea75-4f23-ae41-a85adacc2826-metrics-cert\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:54.002872 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:54.002849    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlsp8\" (UniqueName: \"kubernetes.io/projected/f461e503-ea75-4f23-ae41-a85adacc2826-kube-api-access-rlsp8\") pod \"lws-controller-manager-8994644bb-4xntv\" (UID: \"f461e503-ea75-4f23-ae41-a85adacc2826\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:54.070874 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:54.070838    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:54.202894 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:54.202866    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"]
Jul 28 17:24:54.205151 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:24:54.205120    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf461e503_ea75_4f23_ae41_a85adacc2826.slice/crio-aae8cbc532d288ff3cbd07d45f79c3f88e431121efdde897ca41af91faaaa27f WatchSource:0}: Error finding container aae8cbc532d288ff3cbd07d45f79c3f88e431121efdde897ca41af91faaaa27f: Status 404 returned error can't find the container with id aae8cbc532d288ff3cbd07d45f79c3f88e431121efdde897ca41af91faaaa27f
Jul 28 17:24:55.113881 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:55.113836    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv" event={"ID":"f461e503-ea75-4f23-ae41-a85adacc2826","Type":"ContainerStarted","Data":"aae8cbc532d288ff3cbd07d45f79c3f88e431121efdde897ca41af91faaaa27f"}
Jul 28 17:24:57.123385 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:57.123352    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv" event={"ID":"f461e503-ea75-4f23-ae41-a85adacc2826","Type":"ContainerStarted","Data":"d4354f556bd28e0abd23d4a9d88607ea569643dc704a3add76b6d2b9ed8a0a42"}
Jul 28 17:24:57.123920 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:57.123500    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:24:57.141457 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:24:57.141413    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv" podStartSLOduration=1.384852794 podStartE2EDuration="4.141399598s" totalImagesPullingTime="2.756546804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:57.140384355 +0000 UTC m=+482.483976632" watchObservedRunningTime="2026-07-28 17:24:57.141399598 +0000 UTC m=+482.484991875"
Jul 28 17:25:08.130323 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:08.130294    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8994644bb-4xntv"
Jul 28 17:25:16.212262 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.212230    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"]
Jul 28 17:25:16.215508 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.215491    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.218106 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.218080    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 17:25:16.218304 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.218285    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 17:25:16.219278 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.219258    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cbt59\""
Jul 28 17:25:16.219396 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.219279    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 17:25:16.219396 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.219324    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 17:25:16.226051 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.226028    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"]
Jul 28 17:25:16.373590 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.373532    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4449761e-d0aa-46cd-97ff-c2d1639e37a3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.373759 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.373632    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4449761e-d0aa-46cd-97ff-c2d1639e37a3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.373759 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.373675    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6phjp\" (UniqueName: \"kubernetes.io/projected/4449761e-d0aa-46cd-97ff-c2d1639e37a3-kube-api-access-6phjp\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.474812 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.474731    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4449761e-d0aa-46cd-97ff-c2d1639e37a3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.474812 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.474777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4449761e-d0aa-46cd-97ff-c2d1639e37a3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.474812 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.474811    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phjp\" (UniqueName: \"kubernetes.io/projected/4449761e-d0aa-46cd-97ff-c2d1639e37a3-kube-api-access-6phjp\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.475523 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.475502    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4449761e-d0aa-46cd-97ff-c2d1639e37a3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.477138 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.477116    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4449761e-d0aa-46cd-97ff-c2d1639e37a3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.483066 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.483037    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6phjp\" (UniqueName: \"kubernetes.io/projected/4449761e-d0aa-46cd-97ff-c2d1639e37a3-kube-api-access-6phjp\") pod \"kuadrant-console-plugin-598f86468b-vrjg8\" (UID: \"4449761e-d0aa-46cd-97ff-c2d1639e37a3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.524957 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.524927    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"
Jul 28 17:25:16.650590 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:16.650545    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8"]
Jul 28 17:25:16.653207 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:25:16.653181    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4449761e_d0aa_46cd_97ff_c2d1639e37a3.slice/crio-8b210c64a6041d76c4d9afd63b546536cd7918b5fdab944d9a3e138e93025d9b WatchSource:0}: Error finding container 8b210c64a6041d76c4d9afd63b546536cd7918b5fdab944d9a3e138e93025d9b: Status 404 returned error can't find the container with id 8b210c64a6041d76c4d9afd63b546536cd7918b5fdab944d9a3e138e93025d9b
Jul 28 17:25:17.193998 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:17.193953    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8" event={"ID":"4449761e-d0aa-46cd-97ff-c2d1639e37a3","Type":"ContainerStarted","Data":"8b210c64a6041d76c4d9afd63b546536cd7918b5fdab944d9a3e138e93025d9b"}
Jul 28 17:25:21.211121 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:21.211087    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8" event={"ID":"4449761e-d0aa-46cd-97ff-c2d1639e37a3","Type":"ContainerStarted","Data":"3a63c892ed078028d5dd4c0f5bc5f73a4b5bf41868985acfe65a69ab073d4612"}
Jul 28 17:25:21.230810 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:25:21.230762    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrjg8" podStartSLOduration=0.746059162 podStartE2EDuration="5.230745982s" totalImagesPullingTime="4.48468682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:21.229269419 +0000 UTC m=+506.572861696" watchObservedRunningTime="2026-07-28 17:25:21.230745982 +0000 UTC m=+506.574338259"
Jul 28 17:26:01.471039 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.470957    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-jtvq7"]
Jul 28 17:26:01.489715 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.489688    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-jtvq7"]
Jul 28 17:26:01.489858 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.489779    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-jtvq7"
Jul 28 17:26:01.492909 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.492889    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jwzhh\""
Jul 28 17:26:01.526922 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.526890    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7cz\" (UniqueName: \"kubernetes.io/projected/b1a1c785-6a3c-43fd-8314-91a177063e5d-kube-api-access-kq7cz\") pod \"authorino-b54df94b4-jtvq7\" (UID: \"b1a1c785-6a3c-43fd-8314-91a177063e5d\") " pod="kuadrant-system/authorino-b54df94b4-jtvq7"
Jul 28 17:26:01.628167 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.628125    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7cz\" (UniqueName: \"kubernetes.io/projected/b1a1c785-6a3c-43fd-8314-91a177063e5d-kube-api-access-kq7cz\") pod \"authorino-b54df94b4-jtvq7\" (UID: \"b1a1c785-6a3c-43fd-8314-91a177063e5d\") " pod="kuadrant-system/authorino-b54df94b4-jtvq7"
Jul 28 17:26:01.637600 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.637569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7cz\" (UniqueName: \"kubernetes.io/projected/b1a1c785-6a3c-43fd-8314-91a177063e5d-kube-api-access-kq7cz\") pod \"authorino-b54df94b4-jtvq7\" (UID: \"b1a1c785-6a3c-43fd-8314-91a177063e5d\") " pod="kuadrant-system/authorino-b54df94b4-jtvq7"
Jul 28 17:26:01.798321 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.798291    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-jtvq7"
Jul 28 17:26:01.924317 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:01.924261    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-jtvq7"]
Jul 28 17:26:01.926951 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:26:01.926925    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1a1c785_6a3c_43fd_8314_91a177063e5d.slice/crio-f0ec21d45d58f83fac5edc36cf87e0ef39d09773ea160d07ae35bb2db3913e95 WatchSource:0}: Error finding container f0ec21d45d58f83fac5edc36cf87e0ef39d09773ea160d07ae35bb2db3913e95: Status 404 returned error can't find the container with id f0ec21d45d58f83fac5edc36cf87e0ef39d09773ea160d07ae35bb2db3913e95
Jul 28 17:26:02.353754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:02.353712    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-jtvq7" event={"ID":"b1a1c785-6a3c-43fd-8314-91a177063e5d","Type":"ContainerStarted","Data":"f0ec21d45d58f83fac5edc36cf87e0ef39d09773ea160d07ae35bb2db3913e95"}
Jul 28 17:26:04.916329 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:04.916286    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-jtvq7"]
Jul 28 17:26:05.368521 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:05.368489    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-jtvq7" event={"ID":"b1a1c785-6a3c-43fd-8314-91a177063e5d","Type":"ContainerStarted","Data":"56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666"}
Jul 28 17:26:05.368692 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:05.368574    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-jtvq7" podUID="b1a1c785-6a3c-43fd-8314-91a177063e5d" containerName="authorino" containerID="cri-o://56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666" gracePeriod=30
Jul 28 17:26:05.386613 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:05.386570    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-jtvq7" podStartSLOduration=1.296723653 podStartE2EDuration="4.386537156s" totalImagesPullingTime="3.089813503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:05.385345309 +0000 UTC m=+550.728937585" watchObservedRunningTime="2026-07-28 17:26:05.386537156 +0000 UTC m=+550.730129432"
Jul 28 17:26:05.610231 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:05.610199    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-jtvq7"
Jul 28 17:26:05.661371 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:05.661293    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq7cz\" (UniqueName: \"kubernetes.io/projected/b1a1c785-6a3c-43fd-8314-91a177063e5d-kube-api-access-kq7cz\") pod \"b1a1c785-6a3c-43fd-8314-91a177063e5d\" (UID: \"b1a1c785-6a3c-43fd-8314-91a177063e5d\") "
Jul 28 17:26:05.663410 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:05.663387    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a1c785-6a3c-43fd-8314-91a177063e5d-kube-api-access-kq7cz" (OuterVolumeSpecName: "kube-api-access-kq7cz") pod "b1a1c785-6a3c-43fd-8314-91a177063e5d" (UID: "b1a1c785-6a3c-43fd-8314-91a177063e5d"). InnerVolumeSpecName "kube-api-access-kq7cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:05.762049 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:05.762013    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7cz\" (UniqueName: \"kubernetes.io/projected/b1a1c785-6a3c-43fd-8314-91a177063e5d-kube-api-access-kq7cz\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:26:06.372972 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.372941    2585 generic.go:356] "Generic (PLEG): container finished" podID="b1a1c785-6a3c-43fd-8314-91a177063e5d" containerID="56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666" exitCode=0
Jul 28 17:26:06.373416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.372993    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-jtvq7"
Jul 28 17:26:06.373416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.373031    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-jtvq7" event={"ID":"b1a1c785-6a3c-43fd-8314-91a177063e5d","Type":"ContainerDied","Data":"56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666"}
Jul 28 17:26:06.373416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.373077    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-jtvq7" event={"ID":"b1a1c785-6a3c-43fd-8314-91a177063e5d","Type":"ContainerDied","Data":"f0ec21d45d58f83fac5edc36cf87e0ef39d09773ea160d07ae35bb2db3913e95"}
Jul 28 17:26:06.373416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.373099    2585 scope.go:117] "RemoveContainer" containerID="56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666"
Jul 28 17:26:06.381281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.381260    2585 scope.go:117] "RemoveContainer" containerID="56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666"
Jul 28 17:26:06.381544 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:26:06.381520    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666\": container with ID starting with 56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666 not found: ID does not exist" containerID="56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666"
Jul 28 17:26:06.381668 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.381564    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666"} err="failed to get container status \"56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666\": rpc error: code = NotFound desc = could not find container \"56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666\": container with ID starting with 56a7cc74ab7afafe01bfbef34adfb972855e4bc5502ee4a09775f7e44917c666 not found: ID does not exist"
Jul 28 17:26:06.393621 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.393596    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-jtvq7"]
Jul 28 17:26:06.399589 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:06.399568    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-jtvq7"]
Jul 28 17:26:07.179683 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:07.179650    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a1c785-6a3c-43fd-8314-91a177063e5d" path="/var/lib/kubelet/pods/b1a1c785-6a3c-43fd-8314-91a177063e5d/volumes"
Jul 28 17:26:24.418602 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.418563    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-85tfv"]
Jul 28 17:26:24.418978 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.418832    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a1c785-6a3c-43fd-8314-91a177063e5d" containerName="authorino"
Jul 28 17:26:24.418978 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.418842    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a1c785-6a3c-43fd-8314-91a177063e5d" containerName="authorino"
Jul 28 17:26:24.418978 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.418901    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a1c785-6a3c-43fd-8314-91a177063e5d" containerName="authorino"
Jul 28 17:26:24.421751 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.421734    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.425768 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.425747    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jwzhh\""
Jul 28 17:26:24.425842 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.425747    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 17:26:24.432052 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.432031    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-85tfv"]
Jul 28 17:26:24.514905 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.514876    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk798\" (UniqueName: \"kubernetes.io/projected/417fb782-62e8-4d22-bb74-0105b1d425c6-kube-api-access-gk798\") pod \"authorino-7cf4765b85-85tfv\" (UID: \"417fb782-62e8-4d22-bb74-0105b1d425c6\") " pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.515075 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.514932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/417fb782-62e8-4d22-bb74-0105b1d425c6-tls-cert\") pod \"authorino-7cf4765b85-85tfv\" (UID: \"417fb782-62e8-4d22-bb74-0105b1d425c6\") " pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.616112 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.616071    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/417fb782-62e8-4d22-bb74-0105b1d425c6-tls-cert\") pod \"authorino-7cf4765b85-85tfv\" (UID: \"417fb782-62e8-4d22-bb74-0105b1d425c6\") " pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.616309 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.616151    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk798\" (UniqueName: \"kubernetes.io/projected/417fb782-62e8-4d22-bb74-0105b1d425c6-kube-api-access-gk798\") pod \"authorino-7cf4765b85-85tfv\" (UID: \"417fb782-62e8-4d22-bb74-0105b1d425c6\") " pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.618507 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.618482    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/417fb782-62e8-4d22-bb74-0105b1d425c6-tls-cert\") pod \"authorino-7cf4765b85-85tfv\" (UID: \"417fb782-62e8-4d22-bb74-0105b1d425c6\") " pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.625221 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.625198    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk798\" (UniqueName: \"kubernetes.io/projected/417fb782-62e8-4d22-bb74-0105b1d425c6-kube-api-access-gk798\") pod \"authorino-7cf4765b85-85tfv\" (UID: \"417fb782-62e8-4d22-bb74-0105b1d425c6\") " pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.730509 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.730409    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-85tfv"
Jul 28 17:26:24.858387 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:24.858361    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-85tfv"]
Jul 28 17:26:24.860157 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:26:24.860129    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod417fb782_62e8_4d22_bb74_0105b1d425c6.slice/crio-92723f3e93d93b9b1a24c8155e511d791ab9485def8f572f68febbc0de6851fa WatchSource:0}: Error finding container 92723f3e93d93b9b1a24c8155e511d791ab9485def8f572f68febbc0de6851fa: Status 404 returned error can't find the container with id 92723f3e93d93b9b1a24c8155e511d791ab9485def8f572f68febbc0de6851fa
Jul 28 17:26:25.444649 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:25.444612    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-85tfv" event={"ID":"417fb782-62e8-4d22-bb74-0105b1d425c6","Type":"ContainerStarted","Data":"92723f3e93d93b9b1a24c8155e511d791ab9485def8f572f68febbc0de6851fa"}
Jul 28 17:26:26.449932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:26.449900    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-85tfv" event={"ID":"417fb782-62e8-4d22-bb74-0105b1d425c6","Type":"ContainerStarted","Data":"642621630d335a027e54fbe5769ad5aedb26e8c0d3c127572a706c551580e123"}
Jul 28 17:26:26.466917 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:26.466875    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-85tfv" podStartSLOduration=1.619221788 podStartE2EDuration="2.46686132s" totalImagesPullingTime="847.639532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:26.466228562 +0000 UTC m=+571.809820844" watchObservedRunningTime="2026-07-28 17:26:26.46686132 +0000 UTC m=+571.810453618"
Jul 28 17:26:36.375800 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.375765    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"]
Jul 28 17:26:36.376238 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.375973    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm" podUID="273fcc51-0370-4f95-82d2-a49bbe29db93" containerName="manager" containerID="cri-o://4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8" gracePeriod=10
Jul 28 17:26:36.405080 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.405049    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-svxs4"]
Jul 28 17:26:36.409181 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.409163    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.418748 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.418725    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-svxs4"]
Jul 28 17:26:36.506740 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.506712    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdwq\" (UniqueName: \"kubernetes.io/projected/6d705ac4-a201-43fa-abba-168b7af1d0ac-kube-api-access-hjdwq\") pod \"kserve-controller-manager-85d8b8ddfc-svxs4\" (UID: \"6d705ac4-a201-43fa-abba-168b7af1d0ac\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.506867 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.506778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d705ac4-a201-43fa-abba-168b7af1d0ac-cert\") pod \"kserve-controller-manager-85d8b8ddfc-svxs4\" (UID: \"6d705ac4-a201-43fa-abba-168b7af1d0ac\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.608000 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.607967    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdwq\" (UniqueName: \"kubernetes.io/projected/6d705ac4-a201-43fa-abba-168b7af1d0ac-kube-api-access-hjdwq\") pod \"kserve-controller-manager-85d8b8ddfc-svxs4\" (UID: \"6d705ac4-a201-43fa-abba-168b7af1d0ac\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.608124 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.608030    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d705ac4-a201-43fa-abba-168b7af1d0ac-cert\") pod \"kserve-controller-manager-85d8b8ddfc-svxs4\" (UID: \"6d705ac4-a201-43fa-abba-168b7af1d0ac\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.610302 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.610271    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d705ac4-a201-43fa-abba-168b7af1d0ac-cert\") pod \"kserve-controller-manager-85d8b8ddfc-svxs4\" (UID: \"6d705ac4-a201-43fa-abba-168b7af1d0ac\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.617190 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.617167    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdwq\" (UniqueName: \"kubernetes.io/projected/6d705ac4-a201-43fa-abba-168b7af1d0ac-kube-api-access-hjdwq\") pod \"kserve-controller-manager-85d8b8ddfc-svxs4\" (UID: \"6d705ac4-a201-43fa-abba-168b7af1d0ac\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.621489 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.621471    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:26:36.759494 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.759408    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:36.808815 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.808781    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58kwm\" (UniqueName: \"kubernetes.io/projected/273fcc51-0370-4f95-82d2-a49bbe29db93-kube-api-access-58kwm\") pod \"273fcc51-0370-4f95-82d2-a49bbe29db93\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") "
Jul 28 17:26:36.808966 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.808860    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/273fcc51-0370-4f95-82d2-a49bbe29db93-cert\") pod \"273fcc51-0370-4f95-82d2-a49bbe29db93\" (UID: \"273fcc51-0370-4f95-82d2-a49bbe29db93\") "
Jul 28 17:26:36.810981 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.810950    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273fcc51-0370-4f95-82d2-a49bbe29db93-kube-api-access-58kwm" (OuterVolumeSpecName: "kube-api-access-58kwm") pod "273fcc51-0370-4f95-82d2-a49bbe29db93" (UID: "273fcc51-0370-4f95-82d2-a49bbe29db93"). InnerVolumeSpecName "kube-api-access-58kwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:36.811242 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.811221    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273fcc51-0370-4f95-82d2-a49bbe29db93-cert" (OuterVolumeSpecName: "cert") pod "273fcc51-0370-4f95-82d2-a49bbe29db93" (UID: "273fcc51-0370-4f95-82d2-a49bbe29db93"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:36.885957 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.885927    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-svxs4"]
Jul 28 17:26:36.888305 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:26:36.888279    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d705ac4_a201_43fa_abba_168b7af1d0ac.slice/crio-20c43472d87db4d39002e03113d2f02b77a9b74bb3df55f1cdabdb8c63165698 WatchSource:0}: Error finding container 20c43472d87db4d39002e03113d2f02b77a9b74bb3df55f1cdabdb8c63165698: Status 404 returned error can't find the container with id 20c43472d87db4d39002e03113d2f02b77a9b74bb3df55f1cdabdb8c63165698
Jul 28 17:26:36.909784 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.909753    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58kwm\" (UniqueName: \"kubernetes.io/projected/273fcc51-0370-4f95-82d2-a49bbe29db93-kube-api-access-58kwm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:26:36.909784 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:36.909778    2585 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/273fcc51-0370-4f95-82d2-a49bbe29db93-cert\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:26:37.492056 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.491995    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4" event={"ID":"6d705ac4-a201-43fa-abba-168b7af1d0ac","Type":"ContainerStarted","Data":"5aa49e28218583629ff3c7c8130160dc8956902f9170fe801f5836ae01be7cac"}
Jul 28 17:26:37.492534 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.492135    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:26:37.492534 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.492184    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4" event={"ID":"6d705ac4-a201-43fa-abba-168b7af1d0ac","Type":"ContainerStarted","Data":"20c43472d87db4d39002e03113d2f02b77a9b74bb3df55f1cdabdb8c63165698"}
Jul 28 17:26:37.493399 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.493372    2585 generic.go:356] "Generic (PLEG): container finished" podID="273fcc51-0370-4f95-82d2-a49bbe29db93" containerID="4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8" exitCode=0
Jul 28 17:26:37.493596 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.493405    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"
Jul 28 17:26:37.493596 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.493411    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm" event={"ID":"273fcc51-0370-4f95-82d2-a49bbe29db93","Type":"ContainerDied","Data":"4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8"}
Jul 28 17:26:37.493596 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.493441    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-dv2cm" event={"ID":"273fcc51-0370-4f95-82d2-a49bbe29db93","Type":"ContainerDied","Data":"3b38a2a46802a9e7178cd78d517d2220e27876cc42403081e4b2365ddc3d57d9"}
Jul 28 17:26:37.493596 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.493468    2585 scope.go:117] "RemoveContainer" containerID="4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8"
Jul 28 17:26:37.501371 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.501349    2585 scope.go:117] "RemoveContainer" containerID="4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8"
Jul 28 17:26:37.501666 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:26:37.501641    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8\": container with ID starting with 4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8 not found: ID does not exist" containerID="4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8"
Jul 28 17:26:37.501754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.501672    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8"} err="failed to get container status \"4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8\": rpc error: code = NotFound desc = could not find container \"4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8\": container with ID starting with 4f75c99ca42b9a7b70bc26e0292c5df998d77cf7fa574be8bfa1dc9631a498c8 not found: ID does not exist"
Jul 28 17:26:37.509800 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.509755    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4" podStartSLOduration=1.018820466 podStartE2EDuration="1.509743661s" totalImagesPullingTime="490.923195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:37.50852179 +0000 UTC m=+582.852114060" watchObservedRunningTime="2026-07-28 17:26:37.509743661 +0000 UTC m=+582.853335938"
Jul 28 17:26:37.523546 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.523513    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"]
Jul 28 17:26:37.536476 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:37.536450    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-dv2cm"]
Jul 28 17:26:39.179369 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:39.179337    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="273fcc51-0370-4f95-82d2-a49bbe29db93" path="/var/lib/kubelet/pods/273fcc51-0370-4f95-82d2-a49bbe29db93/volumes"
Jul 28 17:26:55.038222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:55.038195    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:26:55.039903 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:26:55.039883    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:27:08.504004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:08.503899    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-svxs4"
Jul 28 17:27:26.908501 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.908416    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-jlqc6"]
Jul 28 17:27:26.908940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.908768    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273fcc51-0370-4f95-82d2-a49bbe29db93" containerName="manager"
Jul 28 17:27:26.908940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.908784    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="273fcc51-0370-4f95-82d2-a49bbe29db93" containerName="manager"
Jul 28 17:27:26.908940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.908838    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="273fcc51-0370-4f95-82d2-a49bbe29db93" containerName="manager"
Jul 28 17:27:26.911786 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.911770    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jlqc6"
Jul 28 17:27:26.914802 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.914783    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zqwjr\""
Jul 28 17:27:26.914891 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.914816    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:27:26.920911 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.920888    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jlqc6"]
Jul 28 17:27:26.984265 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.984235    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hljb\" (UniqueName: \"kubernetes.io/projected/8b31043f-82fa-4ce2-8bf1-703ce0eef666-kube-api-access-7hljb\") pod \"s3-init-jlqc6\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") " pod="kserve/s3-init-jlqc6"
Jul 28 17:27:26.984430 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:26.984295    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b31043f-82fa-4ce2-8bf1-703ce0eef666-model-cache\") pod \"s3-init-jlqc6\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") " pod="kserve/s3-init-jlqc6"
Jul 28 17:27:27.085611 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:27.085571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b31043f-82fa-4ce2-8bf1-703ce0eef666-model-cache\") pod \"s3-init-jlqc6\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") " pod="kserve/s3-init-jlqc6"
Jul 28 17:27:27.085776 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:27.085634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hljb\" (UniqueName: \"kubernetes.io/projected/8b31043f-82fa-4ce2-8bf1-703ce0eef666-kube-api-access-7hljb\") pod \"s3-init-jlqc6\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") " pod="kserve/s3-init-jlqc6"
Jul 28 17:27:27.085963 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:27.085942    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b31043f-82fa-4ce2-8bf1-703ce0eef666-model-cache\") pod \"s3-init-jlqc6\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") " pod="kserve/s3-init-jlqc6"
Jul 28 17:27:27.095393 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:27.095365    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hljb\" (UniqueName: \"kubernetes.io/projected/8b31043f-82fa-4ce2-8bf1-703ce0eef666-kube-api-access-7hljb\") pod \"s3-init-jlqc6\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") " pod="kserve/s3-init-jlqc6"
Jul 28 17:27:27.220979 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:27.220904    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jlqc6"
Jul 28 17:27:27.349606 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:27.349583    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jlqc6"]
Jul 28 17:27:27.352122 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:27:27.352096    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b31043f_82fa_4ce2_8bf1_703ce0eef666.slice/crio-4d90400c9c243dc91ebb227931d121987d2e64c80eb44c92f609eca333c8779e WatchSource:0}: Error finding container 4d90400c9c243dc91ebb227931d121987d2e64c80eb44c92f609eca333c8779e: Status 404 returned error can't find the container with id 4d90400c9c243dc91ebb227931d121987d2e64c80eb44c92f609eca333c8779e
Jul 28 17:27:27.685946 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:27.685906    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jlqc6" event={"ID":"8b31043f-82fa-4ce2-8bf1-703ce0eef666","Type":"ContainerStarted","Data":"4d90400c9c243dc91ebb227931d121987d2e64c80eb44c92f609eca333c8779e"}
Jul 28 17:27:31.704323 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:31.704284    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jlqc6" event={"ID":"8b31043f-82fa-4ce2-8bf1-703ce0eef666","Type":"ContainerStarted","Data":"1dee85251117fa5d86cb618ac2d31e6baa25b8d3e55e2da7fd36d7d89f6b8f0b"}
Jul 28 17:27:44.755773 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:44.755677    2585 generic.go:356] "Generic (PLEG): container finished" podID="8b31043f-82fa-4ce2-8bf1-703ce0eef666" containerID="1dee85251117fa5d86cb618ac2d31e6baa25b8d3e55e2da7fd36d7d89f6b8f0b" exitCode=0
Jul 28 17:27:44.755773 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:44.755756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jlqc6" event={"ID":"8b31043f-82fa-4ce2-8bf1-703ce0eef666","Type":"ContainerDied","Data":"1dee85251117fa5d86cb618ac2d31e6baa25b8d3e55e2da7fd36d7d89f6b8f0b"}
Jul 28 17:27:49.778878 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:49.778841    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jlqc6" event={"ID":"8b31043f-82fa-4ce2-8bf1-703ce0eef666","Type":"ContainerStarted","Data":"e328765e7a2b9ba9a95d7854a5f2a5a7c4447f102cf78cbfb34bbb4622163d34"}
Jul 28 17:27:49.799171 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:27:49.799111    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-jlqc6" podStartSLOduration=2.514578814 podStartE2EDuration="23.799090052s" totalImagesPullingTime="7.854617589s" totalInitContainerRuntime="13.429893649s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:49.796386944 +0000 UTC m=+655.139979222" watchObservedRunningTime="2026-07-28 17:27:49.799090052 +0000 UTC m=+655.142682332"
Jul 28 17:28:55.031636 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:55.031603    2585 generic.go:356] "Generic (PLEG): container finished" podID="8b31043f-82fa-4ce2-8bf1-703ce0eef666" containerID="e328765e7a2b9ba9a95d7854a5f2a5a7c4447f102cf78cbfb34bbb4622163d34" exitCode=0
Jul 28 17:28:55.032058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:55.031680    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jlqc6" event={"ID":"8b31043f-82fa-4ce2-8bf1-703ce0eef666","Type":"ContainerDied","Data":"e328765e7a2b9ba9a95d7854a5f2a5a7c4447f102cf78cbfb34bbb4622163d34"}
Jul 28 17:28:56.160230 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:56.160208    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jlqc6"
Jul 28 17:28:56.208951 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:56.208926    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hljb\" (UniqueName: \"kubernetes.io/projected/8b31043f-82fa-4ce2-8bf1-703ce0eef666-kube-api-access-7hljb\") pod \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") "
Jul 28 17:28:56.209064 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:56.208977    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b31043f-82fa-4ce2-8bf1-703ce0eef666-model-cache\") pod \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\" (UID: \"8b31043f-82fa-4ce2-8bf1-703ce0eef666\") "
Jul 28 17:28:56.210918 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:56.210887    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b31043f-82fa-4ce2-8bf1-703ce0eef666-kube-api-access-7hljb" (OuterVolumeSpecName: "kube-api-access-7hljb") pod "8b31043f-82fa-4ce2-8bf1-703ce0eef666" (UID: "8b31043f-82fa-4ce2-8bf1-703ce0eef666"). InnerVolumeSpecName "kube-api-access-7hljb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:28:56.227987 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:56.227946    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b31043f-82fa-4ce2-8bf1-703ce0eef666-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b31043f-82fa-4ce2-8bf1-703ce0eef666" (UID: "8b31043f-82fa-4ce2-8bf1-703ce0eef666"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:28:56.309504 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:56.309438    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hljb\" (UniqueName: \"kubernetes.io/projected/8b31043f-82fa-4ce2-8bf1-703ce0eef666-kube-api-access-7hljb\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:28:56.309504 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:56.309468    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b31043f-82fa-4ce2-8bf1-703ce0eef666-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:28:57.039822 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:57.039793    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jlqc6"
Jul 28 17:28:57.039970 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:57.039793    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jlqc6" event={"ID":"8b31043f-82fa-4ce2-8bf1-703ce0eef666","Type":"ContainerDied","Data":"4d90400c9c243dc91ebb227931d121987d2e64c80eb44c92f609eca333c8779e"}
Jul 28 17:28:57.039970 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:28:57.039907    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d90400c9c243dc91ebb227931d121987d2e64c80eb44c92f609eca333c8779e"
Jul 28 17:29:27.152603 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:27.152542    2585 generic.go:356] "Generic (PLEG): container finished" podID="c4c71c04-b83e-44b3-b3ce-4a06dfec9307" containerID="82fe3ae95ffcbea11d7831a576d0a1f6c1009b3f1c05320e3c452d7641801144" exitCode=0
Jul 28 17:29:27.153027 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:27.152617    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" event={"ID":"c4c71c04-b83e-44b3-b3ce-4a06dfec9307","Type":"ContainerDied","Data":"82fe3ae95ffcbea11d7831a576d0a1f6c1009b3f1c05320e3c452d7641801144"}
Jul 28 17:29:27.153072 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:27.153030    2585 scope.go:117] "RemoveContainer" containerID="82fe3ae95ffcbea11d7831a576d0a1f6c1009b3f1c05320e3c452d7641801144"
Jul 28 17:29:27.153942 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:27.153915    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 17:29:27.153998 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:27.153987    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 17:29:27.154045 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:27.154034    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 17:29:28.159133 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:28.159096    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" event={"ID":"c4c71c04-b83e-44b3-b3ce-4a06dfec9307","Type":"ContainerStarted","Data":"0e86c358777c2171fafc1bd8da5e9f0052f13848fb3a1112216e3bb4d35815d1"}
Jul 28 17:29:28.682723 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:28.682682    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:29:28.683999 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:28.683962    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 28 17:29:28.684136 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:28.684070    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" podUID="c4c71c04-b83e-44b3-b3ce-4a06dfec9307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 28 17:29:29.682896 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:29.682851    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 28 17:29:29.683417 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:29.682919    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" podUID="c4c71c04-b83e-44b3-b3ce-4a06dfec9307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 28 17:29:30.682141 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:30.682092    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 28 17:29:30.682346 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:30.682166    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" podUID="c4c71c04-b83e-44b3-b3ce-4a06dfec9307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 28 17:29:31.682823 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:31.682785    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 28 17:29:31.683406 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:31.683373    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" podUID="c4c71c04-b83e-44b3-b3ce-4a06dfec9307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 28 17:29:32.081960 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:32.081923    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:29:32.682711 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:32.682671    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 28 17:29:32.682915 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:32.682744    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l" podUID="c4c71c04-b83e-44b3-b3ce-4a06dfec9307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 28 17:29:33.685748 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:33.685719    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:29:33.686717 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:33.686697    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l"
Jul 28 17:29:43.493484 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.493444    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"]
Jul 28 17:29:43.493932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.493915    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b31043f-82fa-4ce2-8bf1-703ce0eef666" containerName="download-hf-model"
Jul 28 17:29:43.493979 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.493935    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b31043f-82fa-4ce2-8bf1-703ce0eef666" containerName="download-hf-model"
Jul 28 17:29:43.493979 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.493974    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b31043f-82fa-4ce2-8bf1-703ce0eef666" containerName="s3-init"
Jul 28 17:29:43.494039 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.493983    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b31043f-82fa-4ce2-8bf1-703ce0eef666" containerName="s3-init"
Jul 28 17:29:43.494109 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.494051    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b31043f-82fa-4ce2-8bf1-703ce0eef666" containerName="s3-init"
Jul 28 17:29:43.497281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.497261    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.500499 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.500477    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 28 17:29:43.500633 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.500484    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 28 17:29:43.500721 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.500703    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-65t55\""
Jul 28 17:29:43.500784 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.500770    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 28 17:29:43.500961 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.500946    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 28 17:29:43.513985 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.513964    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"]
Jul 28 17:29:43.601866 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.601837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.601866 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.601869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.602045 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.601896    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72cnn\" (UniqueName: \"kubernetes.io/projected/ec8c770f-4829-4d70-809d-2d31b8297a9a-kube-api-access-72cnn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.602045 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.601945    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8c770f-4829-4d70-809d-2d31b8297a9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.602045 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.601961    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.602045 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.602014    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.602045 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.602036    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.702612 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.702581    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8c770f-4829-4d70-809d-2d31b8297a9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.702746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.702616    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.702746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.702648    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.702746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.702682    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.702746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.702723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.702934 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.702749    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.702934 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.702785    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72cnn\" (UniqueName: \"kubernetes.io/projected/ec8c770f-4829-4d70-809d-2d31b8297a9a-kube-api-access-72cnn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.703144 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.703119    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.703219 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.703157    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.703219 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.703190    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.703290 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.703239    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.704882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.704849    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8c770f-4829-4d70-809d-2d31b8297a9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.705052 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.705034    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8c770f-4829-4d70-809d-2d31b8297a9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.717274 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.717242    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72cnn\" (UniqueName: \"kubernetes.io/projected/ec8c770f-4829-4d70-809d-2d31b8297a9a-kube-api-access-72cnn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" (UID: \"ec8c770f-4829-4d70-809d-2d31b8297a9a\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.809759 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.809731    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"
Jul 28 17:29:43.958544 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:43.958501    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb"]
Jul 28 17:29:43.962072 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:29:43.962028    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8c770f_4829_4d70_809d_2d31b8297a9a.slice/crio-e68b804afbd712e9a4703822ce279dbcb9b4484c24326f361e7b2eb5b889baee WatchSource:0}: Error finding container e68b804afbd712e9a4703822ce279dbcb9b4484c24326f361e7b2eb5b889baee: Status 404 returned error can't find the container with id e68b804afbd712e9a4703822ce279dbcb9b4484c24326f361e7b2eb5b889baee
Jul 28 17:29:44.221094 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:44.220987    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb" event={"ID":"ec8c770f-4829-4d70-809d-2d31b8297a9a","Type":"ContainerStarted","Data":"4fc88d8b9aac6954b199294486888fd60ed0dece9cc27b881e8bf4694e780934"}
Jul 28 17:29:44.221094 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:29:44.221061    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb" event={"ID":"ec8c770f-4829-4d70-809d-2d31b8297a9a","Type":"ContainerStarted","Data":"e68b804afbd712e9a4703822ce279dbcb9b4484c24326f361e7b2eb5b889baee"}
Jul 28 17:31:55.064442 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:31:55.064364    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:31:55.064961 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:31:55.064582    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:36:55.091743 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:36:55.091715    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:36:55.092704 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:36:55.092678    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:41:55.115586 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:41:55.115533    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:41:55.117896 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:41:55.117268    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:44:12.316613 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:12.316519    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-fac1c4ae_llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb_ec8c770f-4829-4d70-809d-2d31b8297a9a/storage-initializer/0.log"
Jul 28 17:44:28.550155 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.550125    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:44:28.553487 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.553465    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.556007 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.555986    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-4fw8t\""
Jul 28 17:44:28.557101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.557075    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 17:44:28.557211 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.557092    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 17:44:28.557211 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.557104    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-wqpph\""
Jul 28 17:44:28.557211 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.557186    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 17:44:28.557211 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.557112    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 17:44:28.572435 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.572415    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:44:28.687453 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.687420    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.687677 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.687467    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.687677 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.687492    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/6959fba9-2e6b-48eb-8336-83b2db66d094-kube-api-access-qrgcc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.687677 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.687516    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6959fba9-2e6b-48eb-8336-83b2db66d094-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.687677 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.687584    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.687677 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.687628    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.687967 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.687684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788383 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788588 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788396    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/6959fba9-2e6b-48eb-8336-83b2db66d094-kube-api-access-qrgcc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788588 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788424    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6959fba9-2e6b-48eb-8336-83b2db66d094-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788588 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788453    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788773 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788773 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788687    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788773 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788736    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788795    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.788899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.789005 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.788989    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.789060 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.789004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.790787 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.790766    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.790872 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.790845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6959fba9-2e6b-48eb-8336-83b2db66d094-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.799945 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.799917    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/6959fba9-2e6b-48eb-8336-83b2db66d094-kube-api-access-qrgcc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:28.864285 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:28.864200    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:44:29.003266 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:29.003237    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:44:29.005147 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:44:29.005124    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6959fba9_2e6b_48eb_8336_83b2db66d094.slice/crio-7018e1aa15f3d23cc0a514379bb7f2700eeeedbb3640a77c06735eb94a49523e WatchSource:0}: Error finding container 7018e1aa15f3d23cc0a514379bb7f2700eeeedbb3640a77c06735eb94a49523e: Status 404 returned error can't find the container with id 7018e1aa15f3d23cc0a514379bb7f2700eeeedbb3640a77c06735eb94a49523e
Jul 28 17:44:29.558217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:29.558176    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6959fba9-2e6b-48eb-8336-83b2db66d094","Type":"ContainerStarted","Data":"f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68"}
Jul 28 17:44:29.558608 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:29.558233    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6959fba9-2e6b-48eb-8336-83b2db66d094","Type":"ContainerStarted","Data":"7018e1aa15f3d23cc0a514379bb7f2700eeeedbb3640a77c06735eb94a49523e"}
Jul 28 17:44:38.600435 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:38.600351    2585 generic.go:356] "Generic (PLEG): container finished" podID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerID="f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68" exitCode=0
Jul 28 17:44:38.600826 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:38.600433    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6959fba9-2e6b-48eb-8336-83b2db66d094","Type":"ContainerDied","Data":"f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68"}
Jul 28 17:44:38.602038 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:44:38.602023    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:45:08.760411 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:08.760378    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6959fba9-2e6b-48eb-8336-83b2db66d094","Type":"ContainerStarted","Data":"d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036"}
Jul 28 17:45:08.786047 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:08.785989    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.041145873 podStartE2EDuration="40.7859698s" totalImagesPullingTime="29.528393438s" totalInitContainerRuntime="9.216430489s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:08.783112829 +0000 UTC m=+1694.126705110" watchObservedRunningTime="2026-07-28 17:45:08.7859698 +0000 UTC m=+1694.129562078"
Jul 28 17:45:08.864411 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:08.864381    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:45:08.864586 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:08.864438    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:45:08.865681 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:08.865651    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:45:18.865374 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:18.865284    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:45:28.864766 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:28.864725    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:45:38.865363 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:38.865312    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:45:48.865381 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:48.865330    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:45:58.865023 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:45:58.864971    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:46:08.865684 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:46:08.865637    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:46:18.865100 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:46:18.865053    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:46:28.865083 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:46:28.865035    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 28 17:46:38.874446 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:46:38.874412    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:46:38.882361 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:46:38.882343    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:46:55.139462 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:46:55.139344    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:46:55.145376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:46:55.142171    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:47:28.309870 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.309832    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"]
Jul 28 17:47:28.313026 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.313007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.316173 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.316139    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 17:47:28.316173 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.316153    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 17:47:28.316346 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.316175    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 28 17:47:28.316346 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.316306    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-j75r7\""
Jul 28 17:47:28.316429 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.316410    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 17:47:28.326795 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.326774    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"]
Jul 28 17:47:28.433850 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.433817    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-model-cache\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.433850 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.433850    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3880937-a906-4f3e-8de5-71b49057f6cb-tls-certs\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.434058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.433872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-home\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.434058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.433910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqwj\" (UniqueName: \"kubernetes.io/projected/e3880937-a906-4f3e-8de5-71b49057f6cb-kube-api-access-qlqwj\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.434058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.433942    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-tmp-dir\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.434058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.433982    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-dshm\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.434058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.434040    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535107 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535075    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-dshm\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535132    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535164    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-model-cache\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3880937-a906-4f3e-8de5-71b49057f6cb-tls-certs\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535213    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-home\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqwj\" (UniqueName: \"kubernetes.io/projected/e3880937-a906-4f3e-8de5-71b49057f6cb-kube-api-access-qlqwj\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535352 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535292    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-tmp-dir\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535698 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535698 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535635    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-model-cache\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535698 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535682    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-home\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.535837 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.535750    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-tmp-dir\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.537491 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.537469    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-dshm\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.537768 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.537750    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3880937-a906-4f3e-8de5-71b49057f6cb-tls-certs\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.546045 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.546025    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqwj\" (UniqueName: \"kubernetes.io/projected/e3880937-a906-4f3e-8de5-71b49057f6cb-kube-api-access-qlqwj\") pod \"auth-invalid-token-test-kserve-7f8566565d-d7bk7\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.623003 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.622931    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:28.765409 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:28.765379    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"]
Jul 28 17:47:28.767643 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:47:28.767613    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3880937_a906_4f3e_8de5_71b49057f6cb.slice/crio-3180772de75f3046759b63453679086543e1fb726ec8ca173e4e2678886c579a WatchSource:0}: Error finding container 3180772de75f3046759b63453679086543e1fb726ec8ca173e4e2678886c579a: Status 404 returned error can't find the container with id 3180772de75f3046759b63453679086543e1fb726ec8ca173e4e2678886c579a
Jul 28 17:47:29.336830 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:29.336797    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" event={"ID":"e3880937-a906-4f3e-8de5-71b49057f6cb","Type":"ContainerStarted","Data":"e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0"}
Jul 28 17:47:29.337293 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:29.336846    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" event={"ID":"e3880937-a906-4f3e-8de5-71b49057f6cb","Type":"ContainerStarted","Data":"3180772de75f3046759b63453679086543e1fb726ec8ca173e4e2678886c579a"}
Jul 28 17:47:34.365949 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:34.365904    2585 generic.go:356] "Generic (PLEG): container finished" podID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerID="e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0" exitCode=0
Jul 28 17:47:34.366404 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:34.365975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" event={"ID":"e3880937-a906-4f3e-8de5-71b49057f6cb","Type":"ContainerDied","Data":"e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0"}
Jul 28 17:47:35.374100 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:35.374058    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" event={"ID":"e3880937-a906-4f3e-8de5-71b49057f6cb","Type":"ContainerStarted","Data":"e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7"}
Jul 28 17:47:35.400409 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:35.400361    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podStartSLOduration=2.835455406 podStartE2EDuration="7.400347185s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.564891779s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:35.397754341 +0000 UTC m=+1840.741346653" watchObservedRunningTime="2026-07-28 17:47:35.400347185 +0000 UTC m=+1840.743939462"
Jul 28 17:47:35.623165 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:35.623130    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:38.623958 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:38.623863    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:47:38.625386 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:38.625356    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:47:40.047424 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:40.047390    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:47:48.623596 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:48.623534    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:47:55.070614 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.070536    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main" containerID="cri-o://d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036" gracePeriod=45
Jul 28 17:47:55.834693 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.834669    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:47:55.884851 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.884771    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-tmp-dir\") pod \"6959fba9-2e6b-48eb-8336-83b2db66d094\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") "
Jul 28 17:47:55.884851 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.884820    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-home\") pod \"6959fba9-2e6b-48eb-8336-83b2db66d094\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") "
Jul 28 17:47:55.885067 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.884857    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-kserve-provision-location\") pod \"6959fba9-2e6b-48eb-8336-83b2db66d094\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") "
Jul 28 17:47:55.885067 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.884904    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-dshm\") pod \"6959fba9-2e6b-48eb-8336-83b2db66d094\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") "
Jul 28 17:47:55.885067 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.884936    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/6959fba9-2e6b-48eb-8336-83b2db66d094-kube-api-access-qrgcc\") pod \"6959fba9-2e6b-48eb-8336-83b2db66d094\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") "
Jul 28 17:47:55.885067 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.884964    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-model-cache\") pod \"6959fba9-2e6b-48eb-8336-83b2db66d094\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") "
Jul 28 17:47:55.885067 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.884997    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6959fba9-2e6b-48eb-8336-83b2db66d094-tls-certs\") pod \"6959fba9-2e6b-48eb-8336-83b2db66d094\" (UID: \"6959fba9-2e6b-48eb-8336-83b2db66d094\") "
Jul 28 17:47:55.885587 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.885528    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-model-cache" (OuterVolumeSpecName: "model-cache") pod "6959fba9-2e6b-48eb-8336-83b2db66d094" (UID: "6959fba9-2e6b-48eb-8336-83b2db66d094"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:55.885690 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.885660    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-home" (OuterVolumeSpecName: "home") pod "6959fba9-2e6b-48eb-8336-83b2db66d094" (UID: "6959fba9-2e6b-48eb-8336-83b2db66d094"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:55.887395 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.887371    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6959fba9-2e6b-48eb-8336-83b2db66d094-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6959fba9-2e6b-48eb-8336-83b2db66d094" (UID: "6959fba9-2e6b-48eb-8336-83b2db66d094"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:55.887567 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.887521    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-dshm" (OuterVolumeSpecName: "dshm") pod "6959fba9-2e6b-48eb-8336-83b2db66d094" (UID: "6959fba9-2e6b-48eb-8336-83b2db66d094"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:55.887647 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.887604    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6959fba9-2e6b-48eb-8336-83b2db66d094-kube-api-access-qrgcc" (OuterVolumeSpecName: "kube-api-access-qrgcc") pod "6959fba9-2e6b-48eb-8336-83b2db66d094" (UID: "6959fba9-2e6b-48eb-8336-83b2db66d094"). InnerVolumeSpecName "kube-api-access-qrgcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:55.898058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.898036    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6959fba9-2e6b-48eb-8336-83b2db66d094" (UID: "6959fba9-2e6b-48eb-8336-83b2db66d094"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:55.946271 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.946233    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6959fba9-2e6b-48eb-8336-83b2db66d094" (UID: "6959fba9-2e6b-48eb-8336-83b2db66d094"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:55.986340 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.986311    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.986340 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.986337    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.986477 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.986346    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.986477 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.986356    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.986477 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.986364    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/6959fba9-2e6b-48eb-8336-83b2db66d094-kube-api-access-qrgcc\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.986477 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.986371    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6959fba9-2e6b-48eb-8336-83b2db66d094-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.986477 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:55.986379    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6959fba9-2e6b-48eb-8336-83b2db66d094-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:47:56.475499 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.475460    2585 generic.go:356] "Generic (PLEG): container finished" podID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerID="d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036" exitCode=0
Jul 28 17:47:56.475914 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.475507    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:47:56.475914 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.475536    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6959fba9-2e6b-48eb-8336-83b2db66d094","Type":"ContainerDied","Data":"d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036"}
Jul 28 17:47:56.475914 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.475603    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6959fba9-2e6b-48eb-8336-83b2db66d094","Type":"ContainerDied","Data":"7018e1aa15f3d23cc0a514379bb7f2700eeeedbb3640a77c06735eb94a49523e"}
Jul 28 17:47:56.475914 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.475627    2585 scope.go:117] "RemoveContainer" containerID="d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036"
Jul 28 17:47:56.484794 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.484778    2585 scope.go:117] "RemoveContainer" containerID="f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68"
Jul 28 17:47:56.494781 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.494761    2585 scope.go:117] "RemoveContainer" containerID="d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036"
Jul 28 17:47:56.495029 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:47:56.495006    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036\": container with ID starting with d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036 not found: ID does not exist" containerID="d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036"
Jul 28 17:47:56.495099 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.495041    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036"} err="failed to get container status \"d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036\": rpc error: code = NotFound desc = could not find container \"d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036\": container with ID starting with d9dc5b91983b2fc7586dfd29616f865bd489c2764f557a15501b3224bf447036 not found: ID does not exist"
Jul 28 17:47:56.495099 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.495067    2585 scope.go:117] "RemoveContainer" containerID="f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68"
Jul 28 17:47:56.495320 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:47:56.495302    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68\": container with ID starting with f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68 not found: ID does not exist" containerID="f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68"
Jul 28 17:47:56.495364 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.495327    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68"} err="failed to get container status \"f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68\": rpc error: code = NotFound desc = could not find container \"f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68\": container with ID starting with f37e8e506bb20c1c98dbf1d392e0d86472645154bf07d062a186095818cbef68 not found: ID does not exist"
Jul 28 17:47:56.499388 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.499367    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:47:56.502972 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:56.502951    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:47:57.181615 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:57.181575    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" path="/var/lib/kubelet/pods/6959fba9-2e6b-48eb-8336-83b2db66d094/volumes"
Jul 28 17:47:58.624346 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:47:58.624305    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:48:08.624393 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:48:08.624340    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:48:18.624032 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:48:18.623936    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:48:28.624322 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:48:28.624268    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:48:38.623773 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:48:38.623659    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:48:48.624092 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:48:48.624050    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:48:58.624040 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:48:58.623989    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:49:08.624077 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:08.624025    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:49:18.624074 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:18.624030    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 28 17:49:28.633525 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:28.633490    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:49:28.641369 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:28.641345    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:49:30.743608 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:30.743575    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"]
Jul 28 17:49:45.766825 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:45.766778    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main" containerID="cri-o://e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7" gracePeriod=45
Jul 28 17:49:46.518707 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.518681    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:49:46.602404 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602307    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-home\") pod \"e3880937-a906-4f3e-8de5-71b49057f6cb\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") "
Jul 28 17:49:46.602404 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602372    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3880937-a906-4f3e-8de5-71b49057f6cb-tls-certs\") pod \"e3880937-a906-4f3e-8de5-71b49057f6cb\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") "
Jul 28 17:49:46.602404 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602397    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-dshm\") pod \"e3880937-a906-4f3e-8de5-71b49057f6cb\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") "
Jul 28 17:49:46.602746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602425    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-kserve-provision-location\") pod \"e3880937-a906-4f3e-8de5-71b49057f6cb\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") "
Jul 28 17:49:46.602746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602453    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlqwj\" (UniqueName: \"kubernetes.io/projected/e3880937-a906-4f3e-8de5-71b49057f6cb-kube-api-access-qlqwj\") pod \"e3880937-a906-4f3e-8de5-71b49057f6cb\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") "
Jul 28 17:49:46.602746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602486    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-tmp-dir\") pod \"e3880937-a906-4f3e-8de5-71b49057f6cb\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") "
Jul 28 17:49:46.602746 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602521    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-model-cache\") pod \"e3880937-a906-4f3e-8de5-71b49057f6cb\" (UID: \"e3880937-a906-4f3e-8de5-71b49057f6cb\") "
Jul 28 17:49:46.602952 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.602935    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3880937-a906-4f3e-8de5-71b49057f6cb" (UID: "e3880937-a906-4f3e-8de5-71b49057f6cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:46.603159 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.603124    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-home" (OuterVolumeSpecName: "home") pod "e3880937-a906-4f3e-8de5-71b49057f6cb" (UID: "e3880937-a906-4f3e-8de5-71b49057f6cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:46.604823 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.604793    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-dshm" (OuterVolumeSpecName: "dshm") pod "e3880937-a906-4f3e-8de5-71b49057f6cb" (UID: "e3880937-a906-4f3e-8de5-71b49057f6cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:46.604967 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.604943    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3880937-a906-4f3e-8de5-71b49057f6cb-kube-api-access-qlqwj" (OuterVolumeSpecName: "kube-api-access-qlqwj") pod "e3880937-a906-4f3e-8de5-71b49057f6cb" (UID: "e3880937-a906-4f3e-8de5-71b49057f6cb"). InnerVolumeSpecName "kube-api-access-qlqwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:49:46.605031 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.605008    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3880937-a906-4f3e-8de5-71b49057f6cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3880937-a906-4f3e-8de5-71b49057f6cb" (UID: "e3880937-a906-4f3e-8de5-71b49057f6cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:49:46.613948 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.613912    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3880937-a906-4f3e-8de5-71b49057f6cb" (UID: "e3880937-a906-4f3e-8de5-71b49057f6cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:46.660118 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.660079    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3880937-a906-4f3e-8de5-71b49057f6cb" (UID: "e3880937-a906-4f3e-8de5-71b49057f6cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:46.703276 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.703244    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.703276 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.703276    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.703416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.703286    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.703416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.703294    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3880937-a906-4f3e-8de5-71b49057f6cb-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.703416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.703302    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.703416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.703310    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3880937-a906-4f3e-8de5-71b49057f6cb-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.703416 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.703319    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlqwj\" (UniqueName: \"kubernetes.io/projected/e3880937-a906-4f3e-8de5-71b49057f6cb-kube-api-access-qlqwj\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.947817 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.947697    2585 generic.go:356] "Generic (PLEG): container finished" podID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerID="e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7" exitCode=0
Jul 28 17:49:46.947817 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.947751    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"
Jul 28 17:49:46.947817 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.947756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" event={"ID":"e3880937-a906-4f3e-8de5-71b49057f6cb","Type":"ContainerDied","Data":"e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7"}
Jul 28 17:49:46.947817 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.947811    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7" event={"ID":"e3880937-a906-4f3e-8de5-71b49057f6cb","Type":"ContainerDied","Data":"3180772de75f3046759b63453679086543e1fb726ec8ca173e4e2678886c579a"}
Jul 28 17:49:46.948413 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.947842    2585 scope.go:117] "RemoveContainer" containerID="e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7"
Jul 28 17:49:46.956669 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.956648    2585 scope.go:117] "RemoveContainer" containerID="e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0"
Jul 28 17:49:46.964322 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.964302    2585 scope.go:117] "RemoveContainer" containerID="e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7"
Jul 28 17:49:46.964646 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:49:46.964619    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7\": container with ID starting with e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7 not found: ID does not exist" containerID="e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7"
Jul 28 17:49:46.964719 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.964661    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7"} err="failed to get container status \"e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7\": rpc error: code = NotFound desc = could not find container \"e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7\": container with ID starting with e85f170790112a28ae6ca07519a4700a2f77df5dfd9876d1ea6b850d39ec3ce7 not found: ID does not exist"
Jul 28 17:49:46.964719 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.964683    2585 scope.go:117] "RemoveContainer" containerID="e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0"
Jul 28 17:49:46.964940 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:49:46.964914    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0\": container with ID starting with e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0 not found: ID does not exist" containerID="e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0"
Jul 28 17:49:46.964982 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.964946    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0"} err="failed to get container status \"e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0\": rpc error: code = NotFound desc = could not find container \"e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0\": container with ID starting with e45eaca69fbf1358e4a91d6aaf8743075da8ca957bc2964ed355fe13bb226ec0 not found: ID does not exist"
Jul 28 17:49:46.969707 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.969683    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"]
Jul 28 17:49:46.972932 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:46.972903    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7f8566565d-d7bk7"]
Jul 28 17:49:47.180222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:49:47.180190    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" path="/var/lib/kubelet/pods/e3880937-a906-4f3e-8de5-71b49057f6cb/volumes"
Jul 28 17:50:13.993671 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993637    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"]
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993938    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993949    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993958    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="storage-initializer"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993966    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="storage-initializer"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993973    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993979    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993993    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="storage-initializer"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.993998    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="storage-initializer"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.994049    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3880937-a906-4f3e-8de5-71b49057f6cb" containerName="main"
Jul 28 17:50:13.994147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.994058    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6959fba9-2e6b-48eb-8336-83b2db66d094" containerName="main"
Jul 28 17:50:13.997050 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:13.997030    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.000241 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.000215    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 17:50:14.000386 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.000317    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 28 17:50:14.000386 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.000360    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-wffsh\""
Jul 28 17:50:14.000516 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.000449    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 17:50:14.000599 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.000581    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 17:50:14.007966 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.007943    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"]
Jul 28 17:50:14.141780 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.141742    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-kserve-provision-location\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.141780 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.141783    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-model-cache\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.141998 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.141850    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98k6\" (UniqueName: \"kubernetes.io/projected/38e26e65-3ba2-446a-b922-a51c0e3bc568-kube-api-access-g98k6\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.141998 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.141877    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-dshm\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.141998 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.141901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-home\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.141998 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.141956    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e26e65-3ba2-446a-b922-a51c0e3bc568-tls-certs\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.141998 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.141981    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-tmp-dir\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.242838 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.242804    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e26e65-3ba2-446a-b922-a51c0e3bc568-tls-certs\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.242838 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.242841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-tmp-dir\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243088 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.242869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-kserve-provision-location\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243088 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-model-cache\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243088 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98k6\" (UniqueName: \"kubernetes.io/projected/38e26e65-3ba2-446a-b922-a51c0e3bc568-kube-api-access-g98k6\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243280 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243089    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-dshm\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243280 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-home\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243280 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243242    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-kserve-provision-location\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243495 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243350    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-tmp-dir\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243495 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-home\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.243801 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.243737    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-model-cache\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.245816 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.245789    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-dshm\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.246233 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.246209    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e26e65-3ba2-446a-b922-a51c0e3bc568-tls-certs\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.252933 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.252905    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98k6\" (UniqueName: \"kubernetes.io/projected/38e26e65-3ba2-446a-b922-a51c0e3bc568-kube-api-access-g98k6\") pod \"auth-disabled-test-kserve-5cfc9dcf49-blp8m\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.307966 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.307924    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:14.454904 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:14.454875    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"]
Jul 28 17:50:14.456941 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:50:14.456904    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e26e65_3ba2_446a_b922_a51c0e3bc568.slice/crio-f0d093604080740c68be30608b2e6664d2c7d69691b99a5201be868bd97e2e84 WatchSource:0}: Error finding container f0d093604080740c68be30608b2e6664d2c7d69691b99a5201be868bd97e2e84: Status 404 returned error can't find the container with id f0d093604080740c68be30608b2e6664d2c7d69691b99a5201be868bd97e2e84
Jul 28 17:50:15.062298 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:15.062255    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" event={"ID":"38e26e65-3ba2-446a-b922-a51c0e3bc568","Type":"ContainerStarted","Data":"28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424"}
Jul 28 17:50:15.062777 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:15.062350    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" event={"ID":"38e26e65-3ba2-446a-b922-a51c0e3bc568","Type":"ContainerStarted","Data":"f0d093604080740c68be30608b2e6664d2c7d69691b99a5201be868bd97e2e84"}
Jul 28 17:50:19.081403 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:19.081370    2585 generic.go:356] "Generic (PLEG): container finished" podID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerID="28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424" exitCode=0
Jul 28 17:50:19.081840 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:19.081447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" event={"ID":"38e26e65-3ba2-446a-b922-a51c0e3bc568","Type":"ContainerDied","Data":"28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424"}
Jul 28 17:50:20.087936 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:20.087905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" event={"ID":"38e26e65-3ba2-446a-b922-a51c0e3bc568","Type":"ContainerStarted","Data":"f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a"}
Jul 28 17:50:20.113526 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:20.113473    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podStartSLOduration=2.66761365 podStartE2EDuration="7.113451338s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.445837688s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:50:20.113152616 +0000 UTC m=+2005.456744893" watchObservedRunningTime="2026-07-28 17:50:20.113451338 +0000 UTC m=+2005.457043616"
Jul 28 17:50:20.309042 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:20.308995    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:24.309038 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:24.308993    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:50:24.310491 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:24.310453    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:50:34.309350 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:34.309295    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:50:44.309007 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:44.308957    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:50:54.308972 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:50:54.308920    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:51:01.354684 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.354618    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"]
Jul 28 17:51:01.357310 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.357286    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.361845 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.361684    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 28 17:51:01.361845 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.361700    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-s649w\""
Jul 28 17:51:01.361845 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.361714    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 28 17:51:01.361845 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.361708    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-fbshb\""
Jul 28 17:51:01.361845 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.361740    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 28 17:51:01.369173 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.369150    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"]
Jul 28 17:51:01.453281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.453241    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c3ef057-6b0a-4af5-9a26-89b304e48338-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.453465 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.453300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9l8l\" (UniqueName: \"kubernetes.io/projected/2c3ef057-6b0a-4af5-9a26-89b304e48338-kube-api-access-c9l8l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.554415 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.554372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9l8l\" (UniqueName: \"kubernetes.io/projected/2c3ef057-6b0a-4af5-9a26-89b304e48338-kube-api-access-c9l8l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.554618 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.554449    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c3ef057-6b0a-4af5-9a26-89b304e48338-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.556963 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.556926    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c3ef057-6b0a-4af5-9a26-89b304e48338-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.563379 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.563200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9l8l\" (UniqueName: \"kubernetes.io/projected/2c3ef057-6b0a-4af5-9a26-89b304e48338-kube-api-access-c9l8l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.669296 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.669213    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:01.801200 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.801024    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"]
Jul 28 17:51:01.804175 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:51:01.804147    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3ef057_6b0a_4af5_9a26_89b304e48338.slice/crio-a790c88f00732a7515ab9dcf57b423ac7fdca41a41219339d74df4de0cad74dd WatchSource:0}: Error finding container a790c88f00732a7515ab9dcf57b423ac7fdca41a41219339d74df4de0cad74dd: Status 404 returned error can't find the container with id a790c88f00732a7515ab9dcf57b423ac7fdca41a41219339d74df4de0cad74dd
Jul 28 17:51:01.806710 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:01.806689    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:51:02.260518 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:02.260479    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj" event={"ID":"2c3ef057-6b0a-4af5-9a26-89b304e48338","Type":"ContainerStarted","Data":"a790c88f00732a7515ab9dcf57b423ac7fdca41a41219339d74df4de0cad74dd"}
Jul 28 17:51:04.309471 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:04.309428    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:51:05.276177 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:05.276138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj" event={"ID":"2c3ef057-6b0a-4af5-9a26-89b304e48338","Type":"ContainerStarted","Data":"c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a"}
Jul 28 17:51:05.276518 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:05.276241    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:05.294679 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:05.294615    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj" podStartSLOduration=1.80543221 podStartE2EDuration="4.294594369s" totalImagesPullingTime="2.489162159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:51:05.29324166 +0000 UTC m=+2050.636833945" watchObservedRunningTime="2026-07-28 17:51:05.294594369 +0000 UTC m=+2050.638186647"
Jul 28 17:51:14.309450 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:14.309398    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:51:24.309381 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:24.309281    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:51:34.308820 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:34.308779    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:51:36.282002 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:36.281970    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:37.656343 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:37.656311    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"]
Jul 28 17:51:44.308498 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:44.308448    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:51:46.281716 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:46.281680    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj" podUID="2c3ef057-6b0a-4af5-9a26-89b304e48338" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:51:52.677510 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:52.677440    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj" podUID="2c3ef057-6b0a-4af5-9a26-89b304e48338" containerName="main" containerID="cri-o://c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a" gracePeriod=45
Jul 28 17:51:52.930603 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:52.930535    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:53.007480 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.007441    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9l8l\" (UniqueName: \"kubernetes.io/projected/2c3ef057-6b0a-4af5-9a26-89b304e48338-kube-api-access-c9l8l\") pod \"2c3ef057-6b0a-4af5-9a26-89b304e48338\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") "
Jul 28 17:51:53.007656 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.007499    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c3ef057-6b0a-4af5-9a26-89b304e48338-tls-certs\") pod \"2c3ef057-6b0a-4af5-9a26-89b304e48338\" (UID: \"2c3ef057-6b0a-4af5-9a26-89b304e48338\") "
Jul 28 17:51:53.009656 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.009621    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3ef057-6b0a-4af5-9a26-89b304e48338-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c3ef057-6b0a-4af5-9a26-89b304e48338" (UID: "2c3ef057-6b0a-4af5-9a26-89b304e48338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:53.009767 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.009632    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3ef057-6b0a-4af5-9a26-89b304e48338-kube-api-access-c9l8l" (OuterVolumeSpecName: "kube-api-access-c9l8l") pod "2c3ef057-6b0a-4af5-9a26-89b304e48338" (UID: "2c3ef057-6b0a-4af5-9a26-89b304e48338"). InnerVolumeSpecName "kube-api-access-c9l8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:53.108415 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.108369    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9l8l\" (UniqueName: \"kubernetes.io/projected/2c3ef057-6b0a-4af5-9a26-89b304e48338-kube-api-access-c9l8l\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.108415 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.108405    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c3ef057-6b0a-4af5-9a26-89b304e48338-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.479032 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.478946    2585 generic.go:356] "Generic (PLEG): container finished" podID="2c3ef057-6b0a-4af5-9a26-89b304e48338" containerID="c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a" exitCode=0
Jul 28 17:51:53.479032 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.478983    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"
Jul 28 17:51:53.479032 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.478993    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj" event={"ID":"2c3ef057-6b0a-4af5-9a26-89b304e48338","Type":"ContainerDied","Data":"c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a"}
Jul 28 17:51:53.479276 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.479035    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj" event={"ID":"2c3ef057-6b0a-4af5-9a26-89b304e48338","Type":"ContainerDied","Data":"a790c88f00732a7515ab9dcf57b423ac7fdca41a41219339d74df4de0cad74dd"}
Jul 28 17:51:53.479276 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.479057    2585 scope.go:117] "RemoveContainer" containerID="c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a"
Jul 28 17:51:53.487412 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.487394    2585 scope.go:117] "RemoveContainer" containerID="c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a"
Jul 28 17:51:53.487790 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:51:53.487769    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a\": container with ID starting with c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a not found: ID does not exist" containerID="c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a"
Jul 28 17:51:53.487873 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.487802    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a"} err="failed to get container status \"c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a\": rpc error: code = NotFound desc = could not find container \"c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a\": container with ID starting with c66f014fb90100248fdeaf73e9a9afeef0ed59229e818380e3df62aca052665a not found: ID does not exist"
Jul 28 17:51:53.496814 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.496790    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"]
Jul 28 17:51:53.500652 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:53.500633    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587pv8vj"]
Jul 28 17:51:54.308483 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:54.308442    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:51:55.165753 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:55.165629    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:51:55.179670 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:55.169027    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:51:55.185810 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:51:55.180410    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c3ef057-6b0a-4af5-9a26-89b304e48338" path="/var/lib/kubelet/pods/2c3ef057-6b0a-4af5-9a26-89b304e48338/volumes"
Jul 28 17:52:04.308786 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:04.308731    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:52:09.701377 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.701339    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"]
Jul 28 17:52:09.701871 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.701816    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3ef057-6b0a-4af5-9a26-89b304e48338" containerName="main"
Jul 28 17:52:09.701871 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.701836    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3ef057-6b0a-4af5-9a26-89b304e48338" containerName="main"
Jul 28 17:52:09.702051 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.701950    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3ef057-6b0a-4af5-9a26-89b304e48338" containerName="main"
Jul 28 17:52:09.707458 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.707433    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:09.710503 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.710477    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 17:52:09.710648 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.710477    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 17:52:09.711849 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.711827    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hdmxk\""
Jul 28 17:52:09.711849 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.711827    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 17:52:09.712123 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.712093    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-v568p\""
Jul 28 17:52:09.717144 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.717122    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"]
Jul 28 17:52:09.753609 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.753529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765c3597-ee46-4b6a-8e5d-d115c70645f5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:09.753796 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.753642    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwlb\" (UniqueName: \"kubernetes.io/projected/765c3597-ee46-4b6a-8e5d-d115c70645f5-kube-api-access-dhwlb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:09.854217 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.854180    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwlb\" (UniqueName: \"kubernetes.io/projected/765c3597-ee46-4b6a-8e5d-d115c70645f5-kube-api-access-dhwlb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:09.854394 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.854275    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765c3597-ee46-4b6a-8e5d-d115c70645f5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:09.856789 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.856756    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765c3597-ee46-4b6a-8e5d-d115c70645f5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:09.863007 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:09.862978    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwlb\" (UniqueName: \"kubernetes.io/projected/765c3597-ee46-4b6a-8e5d-d115c70645f5-kube-api-access-dhwlb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:10.018219 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:10.018189    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:10.169707 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:10.169668    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"]
Jul 28 17:52:10.170717 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:52:10.170684    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765c3597_ee46_4b6a_8e5d_d115c70645f5.slice/crio-f82303ddac143399c34a608b26311bf4905e60544bbbd3405fee4ad50f1b04c5 WatchSource:0}: Error finding container f82303ddac143399c34a608b26311bf4905e60544bbbd3405fee4ad50f1b04c5: Status 404 returned error can't find the container with id f82303ddac143399c34a608b26311bf4905e60544bbbd3405fee4ad50f1b04c5
Jul 28 17:52:10.553710 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:10.553673    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" event={"ID":"765c3597-ee46-4b6a-8e5d-d115c70645f5","Type":"ContainerStarted","Data":"9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53"}
Jul 28 17:52:10.553909 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:10.553729    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" event={"ID":"765c3597-ee46-4b6a-8e5d-d115c70645f5","Type":"ContainerStarted","Data":"f82303ddac143399c34a608b26311bf4905e60544bbbd3405fee4ad50f1b04c5"}
Jul 28 17:52:10.553909 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:10.553830    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:10.579712 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:10.579655    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" podStartSLOduration=1.5796352580000002 podStartE2EDuration="1.579635258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:10.578503897 +0000 UTC m=+2115.922096175" watchObservedRunningTime="2026-07-28 17:52:10.579635258 +0000 UTC m=+2115.923227534"
Jul 28 17:52:14.318133 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:14.318100    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:52:14.325761 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:14.325735    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:52:26.840990 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:26.840943    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"]
Jul 28 17:52:41.559927 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:41.559895    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:41.862825 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:41.862727    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" containerID="cri-o://f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a" gracePeriod=45
Jul 28 17:52:42.319385 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.319337    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 17:52:42.608859 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.608835    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:52:42.695024 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.694986    2585 generic.go:356] "Generic (PLEG): container finished" podID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerID="f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a" exitCode=0
Jul 28 17:52:42.695222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.695032    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"
Jul 28 17:52:42.695222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.695064    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" event={"ID":"38e26e65-3ba2-446a-b922-a51c0e3bc568","Type":"ContainerDied","Data":"f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a"}
Jul 28 17:52:42.695222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.695129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m" event={"ID":"38e26e65-3ba2-446a-b922-a51c0e3bc568","Type":"ContainerDied","Data":"f0d093604080740c68be30608b2e6664d2c7d69691b99a5201be868bd97e2e84"}
Jul 28 17:52:42.695222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.695165    2585 scope.go:117] "RemoveContainer" containerID="f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a"
Jul 28 17:52:42.705599 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.705575    2585 scope.go:117] "RemoveContainer" containerID="28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424"
Jul 28 17:52:42.714437 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.714410    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-kserve-provision-location\") pod \"38e26e65-3ba2-446a-b922-a51c0e3bc568\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") "
Jul 28 17:52:42.714591 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.714482    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-home\") pod \"38e26e65-3ba2-446a-b922-a51c0e3bc568\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") "
Jul 28 17:52:42.714591 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.714519    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e26e65-3ba2-446a-b922-a51c0e3bc568-tls-certs\") pod \"38e26e65-3ba2-446a-b922-a51c0e3bc568\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") "
Jul 28 17:52:42.714591 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.714564    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-model-cache\") pod \"38e26e65-3ba2-446a-b922-a51c0e3bc568\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") "
Jul 28 17:52:42.714591 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.714591    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g98k6\" (UniqueName: \"kubernetes.io/projected/38e26e65-3ba2-446a-b922-a51c0e3bc568-kube-api-access-g98k6\") pod \"38e26e65-3ba2-446a-b922-a51c0e3bc568\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") "
Jul 28 17:52:42.714821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.714623    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-dshm\") pod \"38e26e65-3ba2-446a-b922-a51c0e3bc568\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") "
Jul 28 17:52:42.714821 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.714668    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-tmp-dir\") pod \"38e26e65-3ba2-446a-b922-a51c0e3bc568\" (UID: \"38e26e65-3ba2-446a-b922-a51c0e3bc568\") "
Jul 28 17:52:42.715206 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.715143    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-model-cache" (OuterVolumeSpecName: "model-cache") pod "38e26e65-3ba2-446a-b922-a51c0e3bc568" (UID: "38e26e65-3ba2-446a-b922-a51c0e3bc568"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:42.715376 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.715269    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-home" (OuterVolumeSpecName: "home") pod "38e26e65-3ba2-446a-b922-a51c0e3bc568" (UID: "38e26e65-3ba2-446a-b922-a51c0e3bc568"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:42.715723 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.715631    2585 scope.go:117] "RemoveContainer" containerID="f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a"
Jul 28 17:52:42.716085 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:52:42.716054    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a\": container with ID starting with f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a not found: ID does not exist" containerID="f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a"
Jul 28 17:52:42.716168 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.716101    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a"} err="failed to get container status \"f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a\": rpc error: code = NotFound desc = could not find container \"f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a\": container with ID starting with f73c7ec37d7ed225299aa2afd866492cb44f67e0939c334fbe745c603d6e8c0a not found: ID does not exist"
Jul 28 17:52:42.716168 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.716127    2585 scope.go:117] "RemoveContainer" containerID="28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424"
Jul 28 17:52:42.716511 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:52:42.716487    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424\": container with ID starting with 28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424 not found: ID does not exist" containerID="28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424"
Jul 28 17:52:42.716613 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.716518    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424"} err="failed to get container status \"28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424\": rpc error: code = NotFound desc = could not find container \"28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424\": container with ID starting with 28894318f6791571135fa3ecba937de15d1dd13e67bc20d2e11d68a62bbf4424 not found: ID does not exist"
Jul 28 17:52:42.718012 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.717986    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-dshm" (OuterVolumeSpecName: "dshm") pod "38e26e65-3ba2-446a-b922-a51c0e3bc568" (UID: "38e26e65-3ba2-446a-b922-a51c0e3bc568"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:42.718110 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.718029    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e26e65-3ba2-446a-b922-a51c0e3bc568-kube-api-access-g98k6" (OuterVolumeSpecName: "kube-api-access-g98k6") pod "38e26e65-3ba2-446a-b922-a51c0e3bc568" (UID: "38e26e65-3ba2-446a-b922-a51c0e3bc568"). InnerVolumeSpecName "kube-api-access-g98k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:42.718204 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.718185    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e26e65-3ba2-446a-b922-a51c0e3bc568-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38e26e65-3ba2-446a-b922-a51c0e3bc568" (UID: "38e26e65-3ba2-446a-b922-a51c0e3bc568"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:42.729823 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.729787    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38e26e65-3ba2-446a-b922-a51c0e3bc568" (UID: "38e26e65-3ba2-446a-b922-a51c0e3bc568"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:42.730510 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.730488    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"]
Jul 28 17:52:42.788397 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.788358    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38e26e65-3ba2-446a-b922-a51c0e3bc568" (UID: "38e26e65-3ba2-446a-b922-a51c0e3bc568"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:42.816268 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.816237    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.816268 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.816264    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98k6\" (UniqueName: \"kubernetes.io/projected/38e26e65-3ba2-446a-b922-a51c0e3bc568-kube-api-access-g98k6\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.816268 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.816273    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.816475 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.816282    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.816475 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.816291    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.816475 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.816301    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e26e65-3ba2-446a-b922-a51c0e3bc568-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.816475 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:42.816308    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e26e65-3ba2-446a-b922-a51c0e3bc568-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:43.018163 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:43.018126    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"]
Jul 28 17:52:43.023281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:43.023254    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5cfc9dcf49-blp8m"]
Jul 28 17:52:43.180134 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:43.180092    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" path="/var/lib/kubelet/pods/38e26e65-3ba2-446a-b922-a51c0e3bc568/volumes"
Jul 28 17:52:50.399637 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.399530    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"]
Jul 28 17:52:50.400004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.399834    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="storage-initializer"
Jul 28 17:52:50.400004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.399846    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="storage-initializer"
Jul 28 17:52:50.400004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.399861    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main"
Jul 28 17:52:50.400004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.399867    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main"
Jul 28 17:52:50.400004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.399921    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e26e65-3ba2-446a-b922-a51c0e3bc568" containerName="main"
Jul 28 17:52:50.404505 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.404487    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.412754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.412727    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 28 17:52:50.412881 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.412733    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-epp-sa-dockercfg-q7dcq\""
Jul 28 17:52:50.412881 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.412763    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-dk22b\""
Jul 28 17:52:50.412881 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.412773    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 28 17:52:50.412881 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.412800    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 28 17:52:50.417910 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.417887    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"]
Jul 28 17:52:50.483715 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.483679    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d4cb62-d860-4817-81f3-a3bdb26f689f-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.483851 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.483744    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfcd\" (UniqueName: \"kubernetes.io/projected/63d4cb62-d860-4817-81f3-a3bdb26f689f-kube-api-access-4qfcd\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.584427 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.584390    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d4cb62-d860-4817-81f3-a3bdb26f689f-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.584586 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.584438    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfcd\" (UniqueName: \"kubernetes.io/projected/63d4cb62-d860-4817-81f3-a3bdb26f689f-kube-api-access-4qfcd\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.586882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.586863    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d4cb62-d860-4817-81f3-a3bdb26f689f-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.593367 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.593336    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfcd\" (UniqueName: \"kubernetes.io/projected/63d4cb62-d860-4817-81f3-a3bdb26f689f-kube-api-access-4qfcd\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.714147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.714064    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:50.841230 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:50.841195    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"]
Jul 28 17:52:50.844297 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:52:50.844271    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d4cb62_d860_4817_81f3_a3bdb26f689f.slice/crio-a624a55c154befdcb914627b335761eaafb604b1a54951112851c721fc9a1688 WatchSource:0}: Error finding container a624a55c154befdcb914627b335761eaafb604b1a54951112851c721fc9a1688: Status 404 returned error can't find the container with id a624a55c154befdcb914627b335761eaafb604b1a54951112851c721fc9a1688
Jul 28 17:52:51.560009 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:51.559973    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" podUID="765c3597-ee46-4b6a-8e5d-d115c70645f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:52:51.733580 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:51.733530    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" event={"ID":"63d4cb62-d860-4817-81f3-a3bdb26f689f","Type":"ContainerStarted","Data":"2a10da0b57c1579b0dab49e0e1bfbd6140a39afc653fd97341e87ff0e292dff8"}
Jul 28 17:52:51.733750 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:51.733594    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" event={"ID":"63d4cb62-d860-4817-81f3-a3bdb26f689f","Type":"ContainerStarted","Data":"a624a55c154befdcb914627b335761eaafb604b1a54951112851c721fc9a1688"}
Jul 28 17:52:51.733750 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:51.733616    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:52:51.761371 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:51.761320    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" podStartSLOduration=1.761304583 podStartE2EDuration="1.761304583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:51.760828212 +0000 UTC m=+2157.104420490" watchObservedRunningTime="2026-07-28 17:52:51.761304583 +0000 UTC m=+2157.104896907"
Jul 28 17:52:54.450105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.450065    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 28 17:52:54.450105 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:52:54.450101    2585 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" podUID="765c3597-ee46-4b6a-8e5d-d115c70645f5" containerName="main" containerID="cri-o://9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53"
Jul 28 17:52:54.450531 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.450141    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" podUID="765c3597-ee46-4b6a-8e5d-d115c70645f5" containerName="main" containerID="cri-o://9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53" gracePeriod=49
Jul 28 17:52:54.596717 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.596693    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v_765c3597-ee46-4b6a-8e5d-d115c70645f5/main/0.log"
Jul 28 17:52:54.598961 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.598943    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:54.715931 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.715835    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhwlb\" (UniqueName: \"kubernetes.io/projected/765c3597-ee46-4b6a-8e5d-d115c70645f5-kube-api-access-dhwlb\") pod \"765c3597-ee46-4b6a-8e5d-d115c70645f5\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") "
Jul 28 17:52:54.716088 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.715963    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765c3597-ee46-4b6a-8e5d-d115c70645f5-tls-certs\") pod \"765c3597-ee46-4b6a-8e5d-d115c70645f5\" (UID: \"765c3597-ee46-4b6a-8e5d-d115c70645f5\") "
Jul 28 17:52:54.718140 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.718107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765c3597-ee46-4b6a-8e5d-d115c70645f5-kube-api-access-dhwlb" (OuterVolumeSpecName: "kube-api-access-dhwlb") pod "765c3597-ee46-4b6a-8e5d-d115c70645f5" (UID: "765c3597-ee46-4b6a-8e5d-d115c70645f5"). InnerVolumeSpecName "kube-api-access-dhwlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:54.718264 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.718238    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765c3597-ee46-4b6a-8e5d-d115c70645f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "765c3597-ee46-4b6a-8e5d-d115c70645f5" (UID: "765c3597-ee46-4b6a-8e5d-d115c70645f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:54.745898 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.745873    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v_765c3597-ee46-4b6a-8e5d-d115c70645f5/main/0.log"
Jul 28 17:52:54.745999 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.745913    2585 generic.go:356] "Generic (PLEG): container finished" podID="765c3597-ee46-4b6a-8e5d-d115c70645f5" containerID="9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53" exitCode=1
Jul 28 17:52:54.745999 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.745983    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"
Jul 28 17:52:54.746078 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.746007    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" event={"ID":"765c3597-ee46-4b6a-8e5d-d115c70645f5","Type":"ContainerDied","Data":"9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53"}
Jul 28 17:52:54.746078 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.746065    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v" event={"ID":"765c3597-ee46-4b6a-8e5d-d115c70645f5","Type":"ContainerDied","Data":"f82303ddac143399c34a608b26311bf4905e60544bbbd3405fee4ad50f1b04c5"}
Jul 28 17:52:54.746186 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.746089    2585 scope.go:117] "RemoveContainer" containerID="9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53"
Jul 28 17:52:54.754362 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.754344    2585 scope.go:117] "RemoveContainer" containerID="9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53"
Jul 28 17:52:54.754631 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:52:54.754611    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53\": container with ID starting with 9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53 not found: ID does not exist" containerID="9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53"
Jul 28 17:52:54.754684 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.754639    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53"} err="failed to get container status \"9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53\": rpc error: code = NotFound desc = could not find container \"9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53\": container with ID starting with 9c8293504e9e87d0b32a9726c2c1c952a626942731047927462e7a3023f9ab53 not found: ID does not exist"
Jul 28 17:52:54.773106 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.773079    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"]
Jul 28 17:52:54.777544 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.777523    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4hs6v"]
Jul 28 17:52:54.817281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.817242    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765c3597-ee46-4b6a-8e5d-d115c70645f5-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:54.817281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:54.817274    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwlb\" (UniqueName: \"kubernetes.io/projected/765c3597-ee46-4b6a-8e5d-d115c70645f5-kube-api-access-dhwlb\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:52:55.181686 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:52:55.181654    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="765c3597-ee46-4b6a-8e5d-d115c70645f5" path="/var/lib/kubelet/pods/765c3597-ee46-4b6a-8e5d-d115c70645f5/volumes"
Jul 28 17:53:22.739050 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:53:22.739020    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 17:54:36.444931 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.444897    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"]
Jul 28 17:54:36.445330 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.445202    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="765c3597-ee46-4b6a-8e5d-d115c70645f5" containerName="main"
Jul 28 17:54:36.445330 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.445213    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="765c3597-ee46-4b6a-8e5d-d115c70645f5" containerName="main"
Jul 28 17:54:36.445330 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.445258    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="765c3597-ee46-4b6a-8e5d-d115c70645f5" containerName="main"
Jul 28 17:54:36.448018 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.448001    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.452252 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.452217    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-wd9ww\""
Jul 28 17:54:36.452252 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.452232    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 17:54:36.452438 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.452276    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 17:54:36.452438 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.452232    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 17:54:36.462801 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.462778    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"]
Jul 28 17:54:36.551652 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.551617    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.551652 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.551658    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.551899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.551682    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.551899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.551756    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.551899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.551801    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c440d740-c205-4092-aaa1-8f79c0c05b22-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.551899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.551837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5wp\" (UniqueName: \"kubernetes.io/projected/c440d740-c205-4092-aaa1-8f79c0c05b22-kube-api-access-7x5wp\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653157 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653120    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653350 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653166    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653350 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653218    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653350 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c440d740-c205-4092-aaa1-8f79c0c05b22-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653350 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653288    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5wp\" (UniqueName: \"kubernetes.io/projected/c440d740-c205-4092-aaa1-8f79c0c05b22-kube-api-access-7x5wp\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653601 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653352    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653601 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653520    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653758 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653729    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.653758 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.653753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.655536 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.655516    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.655692 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.655674    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c440d740-c205-4092-aaa1-8f79c0c05b22-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.663662 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.663640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5wp\" (UniqueName: \"kubernetes.io/projected/c440d740-c205-4092-aaa1-8f79c0c05b22-kube-api-access-7x5wp\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.758378 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.758293    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:36.889575 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:36.889157    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"]
Jul 28 17:54:36.897058 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:54:36.897029    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc440d740_c205_4092_aaa1_8f79c0c05b22.slice/crio-9115df26c9f38ab92f987954269b27338b16cb7db6ee23430f08d75934a23f77 WatchSource:0}: Error finding container 9115df26c9f38ab92f987954269b27338b16cb7db6ee23430f08d75934a23f77: Status 404 returned error can't find the container with id 9115df26c9f38ab92f987954269b27338b16cb7db6ee23430f08d75934a23f77
Jul 28 17:54:37.146402 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:37.146367    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" event={"ID":"c440d740-c205-4092-aaa1-8f79c0c05b22","Type":"ContainerStarted","Data":"9115df26c9f38ab92f987954269b27338b16cb7db6ee23430f08d75934a23f77"}
Jul 28 17:54:40.158969 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:40.158928    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" event={"ID":"c440d740-c205-4092-aaa1-8f79c0c05b22","Type":"ContainerStarted","Data":"ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71"}
Jul 28 17:54:40.225020 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:40.224973    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" podStartSLOduration=1.4731742319999999 podStartE2EDuration="4.224958764s" totalImagesPullingTime="2.751784532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:40.222634934 +0000 UTC m=+2265.566227211" watchObservedRunningTime="2026-07-28 17:54:40.224958764 +0000 UTC m=+2265.568551037"
Jul 28 17:54:40.759114 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:40.759076    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:46.758829 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:46.758788    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:46.778474 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:46.778438    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:54:47.201138 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:54:47.201107    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:55:10.681403 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:10.681366    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"]
Jul 28 17:55:10.689971 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:55:10.689938    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:55:10.689971 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:10.689971 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:10.689971 ip-10-0-135-20 kubenswrapper[2585]:  > containerID="ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71" cmd=["/bin/sleep","15"]
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:10.689989    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" message="" err=<
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]:  >
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:55:10.690002    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" podUID="c440d740-c205-4092-aaa1-8f79c0c05b22" containerName="main" containerID="cri-o://ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71"
Jul 28 17:55:10.690175 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:10.690036    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" podUID="c440d740-c205-4092-aaa1-8f79c0c05b22" containerName="main" containerID="cri-o://ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71" gracePeriod=60
Jul 28 17:55:11.037327 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.037301    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:55:11.153941 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.153910    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-home\") pod \"c440d740-c205-4092-aaa1-8f79c0c05b22\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") "
Jul 28 17:55:11.154101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.153954    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-model-cache\") pod \"c440d740-c205-4092-aaa1-8f79c0c05b22\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") "
Jul 28 17:55:11.154101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154014    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-tmp-dir\") pod \"c440d740-c205-4092-aaa1-8f79c0c05b22\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") "
Jul 28 17:55:11.154101 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154069    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x5wp\" (UniqueName: \"kubernetes.io/projected/c440d740-c205-4092-aaa1-8f79c0c05b22-kube-api-access-7x5wp\") pod \"c440d740-c205-4092-aaa1-8f79c0c05b22\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") "
Jul 28 17:55:11.154222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154103    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-dshm\") pod \"c440d740-c205-4092-aaa1-8f79c0c05b22\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") "
Jul 28 17:55:11.154222 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154141    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c440d740-c205-4092-aaa1-8f79c0c05b22-tls-certs\") pod \"c440d740-c205-4092-aaa1-8f79c0c05b22\" (UID: \"c440d740-c205-4092-aaa1-8f79c0c05b22\") "
Jul 28 17:55:11.154324 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154213    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-home" (OuterVolumeSpecName: "home") pod "c440d740-c205-4092-aaa1-8f79c0c05b22" (UID: "c440d740-c205-4092-aaa1-8f79c0c05b22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.154324 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154268    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-model-cache" (OuterVolumeSpecName: "model-cache") pod "c440d740-c205-4092-aaa1-8f79c0c05b22" (UID: "c440d740-c205-4092-aaa1-8f79c0c05b22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.154324 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154280    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c440d740-c205-4092-aaa1-8f79c0c05b22" (UID: "c440d740-c205-4092-aaa1-8f79c0c05b22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.154472 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154414    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.154472 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154434    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.154472 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.154449    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.156179 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.156151    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c440d740-c205-4092-aaa1-8f79c0c05b22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c440d740-c205-4092-aaa1-8f79c0c05b22" (UID: "c440d740-c205-4092-aaa1-8f79c0c05b22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:11.156278 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.156203    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-dshm" (OuterVolumeSpecName: "dshm") pod "c440d740-c205-4092-aaa1-8f79c0c05b22" (UID: "c440d740-c205-4092-aaa1-8f79c0c05b22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.156278 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.156247    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c440d740-c205-4092-aaa1-8f79c0c05b22-kube-api-access-7x5wp" (OuterVolumeSpecName: "kube-api-access-7x5wp") pod "c440d740-c205-4092-aaa1-8f79c0c05b22" (UID: "c440d740-c205-4092-aaa1-8f79c0c05b22"). InnerVolumeSpecName "kube-api-access-7x5wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:11.255146 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.255069    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c440d740-c205-4092-aaa1-8f79c0c05b22-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.255146 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.255098    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5wp\" (UniqueName: \"kubernetes.io/projected/c440d740-c205-4092-aaa1-8f79c0c05b22-kube-api-access-7x5wp\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.255146 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.255109    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c440d740-c205-4092-aaa1-8f79c0c05b22-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.289467 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.289438    2585 generic.go:356] "Generic (PLEG): container finished" podID="c440d740-c205-4092-aaa1-8f79c0c05b22" containerID="ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71" exitCode=0
Jul 28 17:55:11.289618 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.289482    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"
Jul 28 17:55:11.289618 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.289497    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" event={"ID":"c440d740-c205-4092-aaa1-8f79c0c05b22","Type":"ContainerDied","Data":"ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71"}
Jul 28 17:55:11.289618 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.289571    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5" event={"ID":"c440d740-c205-4092-aaa1-8f79c0c05b22","Type":"ContainerDied","Data":"9115df26c9f38ab92f987954269b27338b16cb7db6ee23430f08d75934a23f77"}
Jul 28 17:55:11.289618 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.289609    2585 scope.go:117] "RemoveContainer" containerID="ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71"
Jul 28 17:55:11.298051 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.298034    2585 scope.go:117] "RemoveContainer" containerID="ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71"
Jul 28 17:55:11.298313 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:55:11.298281    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71\": container with ID starting with ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71 not found: ID does not exist" containerID="ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71"
Jul 28 17:55:11.298356 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.298322    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71"} err="failed to get container status \"ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71\": rpc error: code = NotFound desc = could not find container \"ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71\": container with ID starting with ab32d45e9852ab79f8cb659250b8b19e60ce114bd4aa217f77c1f1cf05a60e71 not found: ID does not exist"
Jul 28 17:55:11.308323 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.308301    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"]
Jul 28 17:55:11.314007 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:11.313989    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-lbjg5"]
Jul 28 17:55:13.181133 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:13.181086    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c440d740-c205-4092-aaa1-8f79c0c05b22" path="/var/lib/kubelet/pods/c440d740-c205-4092-aaa1-8f79c0c05b22/volumes"
Jul 28 17:55:50.686498 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.686407    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"]
Jul 28 17:55:50.687193 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.687170    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c440d740-c205-4092-aaa1-8f79c0c05b22" containerName="main"
Jul 28 17:55:50.687291 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.687196    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c440d740-c205-4092-aaa1-8f79c0c05b22" containerName="main"
Jul 28 17:55:50.687291 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.687284    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c440d740-c205-4092-aaa1-8f79c0c05b22" containerName="main"
Jul 28 17:55:50.690332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.690312    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.694589 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.694565    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-7clcb\""
Jul 28 17:55:50.694712 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.694589    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 17:55:50.694712 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.694628    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 17:55:50.694712 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.694594    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 17:55:50.703012 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.702988    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"]
Jul 28 17:55:50.756441 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.756408    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.756634 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.756470    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.756634 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.756544    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.756634 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.756600    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.756776 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.756639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.756776 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.756660    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7trb\" (UniqueName: \"kubernetes.io/projected/eeed6574-a151-48e8-96ab-29bcbbb9a4de-kube-api-access-n7trb\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.857700 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.857660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.857899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.857706    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7trb\" (UniqueName: \"kubernetes.io/projected/eeed6574-a151-48e8-96ab-29bcbbb9a4de-kube-api-access-n7trb\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.857899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.857734    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.857899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.857786    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.857899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.857835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.857899 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.857863    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.858176 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.858067    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.858237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.858201    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.858237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.858201    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.860066 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.860045    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.860253 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.860236    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:50.866875 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:50.866848    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7trb\" (UniqueName: \"kubernetes.io/projected/eeed6574-a151-48e8-96ab-29bcbbb9a4de-kube-api-access-n7trb\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:51.000623 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:51.000516    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:55:51.131052 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:51.131013    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"]
Jul 28 17:55:51.133336 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:55:51.133304    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeed6574_a151_48e8_96ab_29bcbbb9a4de.slice/crio-d668326f4691bc4fc47536373af9ace860af47c0ad6c4a80e9d8640bb01a15bf WatchSource:0}: Error finding container d668326f4691bc4fc47536373af9ace860af47c0ad6c4a80e9d8640bb01a15bf: Status 404 returned error can't find the container with id d668326f4691bc4fc47536373af9ace860af47c0ad6c4a80e9d8640bb01a15bf
Jul 28 17:55:51.455671 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:51.455632    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" event={"ID":"eeed6574-a151-48e8-96ab-29bcbbb9a4de","Type":"ContainerStarted","Data":"b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb"}
Jul 28 17:55:51.455837 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:51.455684    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" event={"ID":"eeed6574-a151-48e8-96ab-29bcbbb9a4de","Type":"ContainerStarted","Data":"d668326f4691bc4fc47536373af9ace860af47c0ad6c4a80e9d8640bb01a15bf"}
Jul 28 17:55:51.477413 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:51.477363    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" podStartSLOduration=1.477344814 podStartE2EDuration="1.477344814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:51.475861693 +0000 UTC m=+2336.819453970" watchObservedRunningTime="2026-07-28 17:55:51.477344814 +0000 UTC m=+2336.820937092"
Jul 28 17:55:52.001237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:55:52.001203    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:56:01.000965 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:01.000904    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:56:01.021121 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:01.021089    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:56:01.505498 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:01.505467    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:56:24.995683 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:24.995643    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"]
Jul 28 17:56:25.004252 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:56:25.004216    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:56:25.004252 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:56:25.004252 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:56:25.004252 ip-10-0-135-20 kubenswrapper[2585]:  > containerID="b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb" cmd=["/bin/sleep","15"]
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.004264    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" message="" err=<
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]:  >
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:56:25.004276    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" podUID="eeed6574-a151-48e8-96ab-29bcbbb9a4de" containerName="main" containerID="cri-o://b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb"
Jul 28 17:56:25.004540 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.004318    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" podUID="eeed6574-a151-48e8-96ab-29bcbbb9a4de" containerName="main" containerID="cri-o://b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb" gracePeriod=60
Jul 28 17:56:25.348473 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.348446    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:56:25.456748 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.456715    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tls-certs\") pod \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") "
Jul 28 17:56:25.456963 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.456773    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-model-cache\") pod \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") "
Jul 28 17:56:25.456963 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.456805    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tmp-dir\") pod \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") "
Jul 28 17:56:25.456963 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.456834    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-dshm\") pod \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") "
Jul 28 17:56:25.456963 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.456863    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7trb\" (UniqueName: \"kubernetes.io/projected/eeed6574-a151-48e8-96ab-29bcbbb9a4de-kube-api-access-n7trb\") pod \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") "
Jul 28 17:56:25.456963 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.456903    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-home\") pod \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\" (UID: \"eeed6574-a151-48e8-96ab-29bcbbb9a4de\") "
Jul 28 17:56:25.457249 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.457109    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-model-cache" (OuterVolumeSpecName: "model-cache") pod "eeed6574-a151-48e8-96ab-29bcbbb9a4de" (UID: "eeed6574-a151-48e8-96ab-29bcbbb9a4de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.457249 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.457176    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eeed6574-a151-48e8-96ab-29bcbbb9a4de" (UID: "eeed6574-a151-48e8-96ab-29bcbbb9a4de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.457368 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.457264    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-home" (OuterVolumeSpecName: "home") pod "eeed6574-a151-48e8-96ab-29bcbbb9a4de" (UID: "eeed6574-a151-48e8-96ab-29bcbbb9a4de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.459174 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.459140    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-dshm" (OuterVolumeSpecName: "dshm") pod "eeed6574-a151-48e8-96ab-29bcbbb9a4de" (UID: "eeed6574-a151-48e8-96ab-29bcbbb9a4de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.459281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.459224    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeed6574-a151-48e8-96ab-29bcbbb9a4de-kube-api-access-n7trb" (OuterVolumeSpecName: "kube-api-access-n7trb") pod "eeed6574-a151-48e8-96ab-29bcbbb9a4de" (UID: "eeed6574-a151-48e8-96ab-29bcbbb9a4de"). InnerVolumeSpecName "kube-api-access-n7trb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:25.459281 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.459227    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeed6574-a151-48e8-96ab-29bcbbb9a4de" (UID: "eeed6574-a151-48e8-96ab-29bcbbb9a4de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:56:25.557754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.557644    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.557754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.557688    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.557754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.557704    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7trb\" (UniqueName: \"kubernetes.io/projected/eeed6574-a151-48e8-96ab-29bcbbb9a4de-kube-api-access-n7trb\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.557754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.557722    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.557754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.557736    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed6574-a151-48e8-96ab-29bcbbb9a4de-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.557754 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.557749    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeed6574-a151-48e8-96ab-29bcbbb9a4de-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.598164 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.598125    2585 generic.go:356] "Generic (PLEG): container finished" podID="eeed6574-a151-48e8-96ab-29bcbbb9a4de" containerID="b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb" exitCode=0
Jul 28 17:56:25.598412 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.598189    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"
Jul 28 17:56:25.598412 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.598216    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" event={"ID":"eeed6574-a151-48e8-96ab-29bcbbb9a4de","Type":"ContainerDied","Data":"b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb"}
Jul 28 17:56:25.598412 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.598269    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf" event={"ID":"eeed6574-a151-48e8-96ab-29bcbbb9a4de","Type":"ContainerDied","Data":"d668326f4691bc4fc47536373af9ace860af47c0ad6c4a80e9d8640bb01a15bf"}
Jul 28 17:56:25.598412 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.598291    2585 scope.go:117] "RemoveContainer" containerID="b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb"
Jul 28 17:56:25.608464 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.608445    2585 scope.go:117] "RemoveContainer" containerID="b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb"
Jul 28 17:56:25.608807 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:56:25.608763    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb\": container with ID starting with b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb not found: ID does not exist" containerID="b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb"
Jul 28 17:56:25.608940 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.608800    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb"} err="failed to get container status \"b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb\": rpc error: code = NotFound desc = could not find container \"b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb\": container with ID starting with b6501fb944a97108886aeeaf5772565f0e87f69883ece22f6cc239c71870d5fb not found: ID does not exist"
Jul 28 17:56:25.639129 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.639101    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"]
Jul 28 17:56:25.648427 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:25.648404    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkc9tf"]
Jul 28 17:56:27.180454 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:27.180423    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeed6574-a151-48e8-96ab-29bcbbb9a4de" path="/var/lib/kubelet/pods/eeed6574-a151-48e8-96ab-29bcbbb9a4de/volumes"
Jul 28 17:56:55.191094 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:55.190992    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:56:55.206072 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:56:55.194589    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 17:57:07.711254 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.711220    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"]
Jul 28 17:57:07.711781 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.711506    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeed6574-a151-48e8-96ab-29bcbbb9a4de" containerName="main"
Jul 28 17:57:07.711781 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.711517    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeed6574-a151-48e8-96ab-29bcbbb9a4de" containerName="main"
Jul 28 17:57:07.711781 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.711608    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeed6574-a151-48e8-96ab-29bcbbb9a4de" containerName="main"
Jul 28 17:57:07.714591 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.714575    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.717739 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.717711    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 17:57:07.717933 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.717917    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-8vd9t\""
Jul 28 17:57:07.718021 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.717968    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 17:57:07.719105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.719080    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 17:57:07.719105 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.719095    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 17:57:07.729234 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.729208    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"]
Jul 28 17:57:07.778505 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.778471    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.778505 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.778508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-home\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.778702 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.778534    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-model-cache\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.778702 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.778612    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a48a03-eb95-4bad-bec4-19c78386d55f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.778702 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.778630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.778702 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.778665    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-dshm\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.778702 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.778679    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhb8\" (UniqueName: \"kubernetes.io/projected/98a48a03-eb95-4bad-bec4-19c78386d55f-kube-api-access-6dhb8\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879199 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879161    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a48a03-eb95-4bad-bec4-19c78386d55f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879199 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879200    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879236    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-dshm\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879257    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhb8\" (UniqueName: \"kubernetes.io/projected/98a48a03-eb95-4bad-bec4-19c78386d55f-kube-api-access-6dhb8\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879284    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-home\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879432 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879346    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-model-cache\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879790 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879766    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879853 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879796    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-model-cache\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879853 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879842    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-home\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.879966 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.879891    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.881595 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.881544    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-dshm\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.881735 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.881719    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a48a03-eb95-4bad-bec4-19c78386d55f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.888744 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.888724    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhb8\" (UniqueName: \"kubernetes.io/projected/98a48a03-eb95-4bad-bec4-19c78386d55f-kube-api-access-6dhb8\") pod \"tokenizer-clean-path-test-kserve-79847955cc-vcl8v\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:07.952053 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.952020    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"]
Jul 28 17:57:07.955794 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.955764    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:07.969256 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:07.969228    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"]
Jul 28 17:57:08.025290 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.025258    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:08.082311 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.082200    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.082311 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.082240    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.082311 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.082281    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.160205 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.160140    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"]
Jul 28 17:57:08.162890 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:57:08.162863    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a48a03_eb95_4bad_bec4_19c78386d55f.slice/crio-4ae7c20bd59722190f741b2f48ff25b90b00a2b9d9e90dce83da95fedab9c0ce WatchSource:0}: Error finding container 4ae7c20bd59722190f741b2f48ff25b90b00a2b9d9e90dce83da95fedab9c0ce: Status 404 returned error can't find the container with id 4ae7c20bd59722190f741b2f48ff25b90b00a2b9d9e90dce83da95fedab9c0ce
Jul 28 17:57:08.183095 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.183073    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.183203 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.183146    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.183203 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.183184    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.183501 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.183482    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.183593 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.183533    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.185332 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.185314    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6544489598-z5f7x\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.267074 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.267041    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:08.397965 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.397937    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"]
Jul 28 17:57:08.400174 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:57:08.400152    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a49b43f_a216_4acc_b4ff_e4ff1240fa92.slice/crio-8b13e0a61d07831522e72fc76467eddf2cce3c3d09aebfeb22cd06c684058976 WatchSource:0}: Error finding container 8b13e0a61d07831522e72fc76467eddf2cce3c3d09aebfeb22cd06c684058976: Status 404 returned error can't find the container with id 8b13e0a61d07831522e72fc76467eddf2cce3c3d09aebfeb22cd06c684058976
Jul 28 17:57:08.771856 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.771816    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" event={"ID":"6a49b43f-a216-4acc-b4ff-e4ff1240fa92","Type":"ContainerStarted","Data":"eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec"}
Jul 28 17:57:08.772275 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.771882    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" event={"ID":"6a49b43f-a216-4acc-b4ff-e4ff1240fa92","Type":"ContainerStarted","Data":"8b13e0a61d07831522e72fc76467eddf2cce3c3d09aebfeb22cd06c684058976"}
Jul 28 17:57:08.773634 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.773601    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" event={"ID":"98a48a03-eb95-4bad-bec4-19c78386d55f","Type":"ContainerStarted","Data":"0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861"}
Jul 28 17:57:08.773780 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:08.773646    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" event={"ID":"98a48a03-eb95-4bad-bec4-19c78386d55f","Type":"ContainerStarted","Data":"4ae7c20bd59722190f741b2f48ff25b90b00a2b9d9e90dce83da95fedab9c0ce"}
Jul 28 17:57:12.796111 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:12.796074    2585 generic.go:356] "Generic (PLEG): container finished" podID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerID="eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec" exitCode=0
Jul 28 17:57:12.796676 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:12.796151    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" event={"ID":"6a49b43f-a216-4acc-b4ff-e4ff1240fa92","Type":"ContainerDied","Data":"eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec"}
Jul 28 17:57:12.797566 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:12.797526    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:57:13.807109 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:13.807061    2585 generic.go:356] "Generic (PLEG): container finished" podID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerID="0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861" exitCode=0
Jul 28 17:57:13.807612 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:13.807168    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" event={"ID":"98a48a03-eb95-4bad-bec4-19c78386d55f","Type":"ContainerDied","Data":"0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861"}
Jul 28 17:57:14.819169 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:14.815628    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" event={"ID":"98a48a03-eb95-4bad-bec4-19c78386d55f","Type":"ContainerStarted","Data":"59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b"}
Jul 28 17:57:14.839900 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:14.839841    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" podStartSLOduration=2.575447948 podStartE2EDuration="7.83982233s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.264374382s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:14.839310136 +0000 UTC m=+2420.182902415" watchObservedRunningTime="2026-07-28 17:57:14.83982233 +0000 UTC m=+2420.183414606"
Jul 28 17:57:15.026423 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:15.026381    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:18.026438 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:18.026164    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:18.046208 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:18.046175    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:18.845589 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:18.845559    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:57:47.978861 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:47.978828    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" event={"ID":"6a49b43f-a216-4acc-b4ff-e4ff1240fa92","Type":"ContainerStarted","Data":"7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad"}
Jul 28 17:57:47.979408 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:47.979083    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:57:47.999310 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:57:47.999263    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" podStartSLOduration=1.8407089330000002 podStartE2EDuration="40.99924817s" totalImagesPullingTime="35.050269713s" totalInitContainerRuntime="4.108269524s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:47.997411702 +0000 UTC m=+2453.341003993" watchObservedRunningTime="2026-07-28 17:57:47.99924817 +0000 UTC m=+2453.342840446"
Jul 28 17:58:18.995389 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:18.995356    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:58:20.053104 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:20.053078    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6544489598-z5f7x_6a49b43f-a216-4acc-b4ff-e4ff1240fa92/main/0.log"
Jul 28 17:58:20.665608 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:20.665570    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"]
Jul 28 17:58:20.665846 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:20.665821    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" podUID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerName="main" containerID="cri-o://7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad" gracePeriod=30
Jul 28 17:58:20.672520 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:20.672492    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"]
Jul 28 17:58:20.681337 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:58:20.681299    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:58:20.681337 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:58:20.681337 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:58:20.681337 ip-10-0-135-20 kubenswrapper[2585]:  > containerID="59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b" cmd=["/bin/sleep","15"]
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:20.681361    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" message="" err=<
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]:  >
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:58:20.681377    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="main" containerID="cri-o://59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b"
Jul 28 17:58:20.681573 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:20.681431    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="main" containerID="cri-o://59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b" gracePeriod=60
Jul 28 17:58:20.834924 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:20.834886    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 28 17:58:21.064467 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.064439    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:58:21.124850 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.124807    2585 generic.go:356] "Generic (PLEG): container finished" podID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerID="59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b" exitCode=0
Jul 28 17:58:21.125033 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.124913    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" event={"ID":"98a48a03-eb95-4bad-bec4-19c78386d55f","Type":"ContainerDied","Data":"59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b"}
Jul 28 17:58:21.125033 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.124967    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v" event={"ID":"98a48a03-eb95-4bad-bec4-19c78386d55f","Type":"ContainerDied","Data":"4ae7c20bd59722190f741b2f48ff25b90b00a2b9d9e90dce83da95fedab9c0ce"}
Jul 28 17:58:21.125033 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.124999    2585 scope.go:117] "RemoveContainer" containerID="59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b"
Jul 28 17:58:21.125212 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.125168    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.129605    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-kserve-provision-location\") pod \"98a48a03-eb95-4bad-bec4-19c78386d55f\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") "
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.129686    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-model-cache\") pod \"98a48a03-eb95-4bad-bec4-19c78386d55f\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") "
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.129715    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a48a03-eb95-4bad-bec4-19c78386d55f-tls-certs\") pod \"98a48a03-eb95-4bad-bec4-19c78386d55f\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") "
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.129766    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-tmp-dir\") pod \"98a48a03-eb95-4bad-bec4-19c78386d55f\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") "
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.129797    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-dshm\") pod \"98a48a03-eb95-4bad-bec4-19c78386d55f\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") "
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.129977    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-model-cache" (OuterVolumeSpecName: "model-cache") pod "98a48a03-eb95-4bad-bec4-19c78386d55f" (UID: "98a48a03-eb95-4bad-bec4-19c78386d55f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.130030    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dhb8\" (UniqueName: \"kubernetes.io/projected/98a48a03-eb95-4bad-bec4-19c78386d55f-kube-api-access-6dhb8\") pod \"98a48a03-eb95-4bad-bec4-19c78386d55f\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") "
Jul 28 17:58:21.130130 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.130074    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-home\") pod \"98a48a03-eb95-4bad-bec4-19c78386d55f\" (UID: \"98a48a03-eb95-4bad-bec4-19c78386d55f\") "
Jul 28 17:58:21.130740 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.130339    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.130740 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.130396    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-home" (OuterVolumeSpecName: "home") pod "98a48a03-eb95-4bad-bec4-19c78386d55f" (UID: "98a48a03-eb95-4bad-bec4-19c78386d55f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:21.130740 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.130433    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98a48a03-eb95-4bad-bec4-19c78386d55f" (UID: "98a48a03-eb95-4bad-bec4-19c78386d55f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:21.135209 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.135176    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a48a03-eb95-4bad-bec4-19c78386d55f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98a48a03-eb95-4bad-bec4-19c78386d55f" (UID: "98a48a03-eb95-4bad-bec4-19c78386d55f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:21.138792 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.138755    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-dshm" (OuterVolumeSpecName: "dshm") pod "98a48a03-eb95-4bad-bec4-19c78386d55f" (UID: "98a48a03-eb95-4bad-bec4-19c78386d55f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:21.138906 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.138818    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a48a03-eb95-4bad-bec4-19c78386d55f-kube-api-access-6dhb8" (OuterVolumeSpecName: "kube-api-access-6dhb8") pod "98a48a03-eb95-4bad-bec4-19c78386d55f" (UID: "98a48a03-eb95-4bad-bec4-19c78386d55f"). InnerVolumeSpecName "kube-api-access-6dhb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:21.143144 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.141670    2585 scope.go:117] "RemoveContainer" containerID="0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861"
Jul 28 17:58:21.159560 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.159480    2585 scope.go:117] "RemoveContainer" containerID="59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b"
Jul 28 17:58:21.159937 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:58:21.159915    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b\": container with ID starting with 59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b not found: ID does not exist" containerID="59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b"
Jul 28 17:58:21.160031 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.159947    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b"} err="failed to get container status \"59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b\": rpc error: code = NotFound desc = could not find container \"59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b\": container with ID starting with 59ae250e6377582d09e6336dc4b27732dd928026c9dc75ca1d380d9fff4d799b not found: ID does not exist"
Jul 28 17:58:21.160031 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.159967    2585 scope.go:117] "RemoveContainer" containerID="0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861"
Jul 28 17:58:21.160259 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:58:21.160231    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861\": container with ID starting with 0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861 not found: ID does not exist" containerID="0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861"
Jul 28 17:58:21.160327 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.160272    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861"} err="failed to get container status \"0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861\": rpc error: code = NotFound desc = could not find container \"0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861\": container with ID starting with 0d7701b673a6ba83425f94594fbb2a88afdc7fa6f4edccbeafe249e324bbf861 not found: ID does not exist"
Jul 28 17:58:21.215616 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.215540    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98a48a03-eb95-4bad-bec4-19c78386d55f" (UID: "98a48a03-eb95-4bad-bec4-19c78386d55f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:21.230775 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.230671    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a48a03-eb95-4bad-bec4-19c78386d55f-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.230775 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.230700    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.230775 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.230713    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.230775 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.230726    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhb8\" (UniqueName: \"kubernetes.io/projected/98a48a03-eb95-4bad-bec4-19c78386d55f-kube-api-access-6dhb8\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.230775 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.230739    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.230775 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.230752    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a48a03-eb95-4bad-bec4-19c78386d55f-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.356765 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.356737    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:58:21.433013 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.432979    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tls-certs\") pod \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") "
Jul 28 17:58:21.433239 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.433028    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tmp-dir\") pod \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") "
Jul 28 17:58:21.433239 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.433125    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-kserve-provision-location\") pod \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\" (UID: \"6a49b43f-a216-4acc-b4ff-e4ff1240fa92\") "
Jul 28 17:58:21.434095 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.433315    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a49b43f-a216-4acc-b4ff-e4ff1240fa92" (UID: "6a49b43f-a216-4acc-b4ff-e4ff1240fa92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:21.434095 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.433401    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.435368 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.435339    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a49b43f-a216-4acc-b4ff-e4ff1240fa92" (UID: "6a49b43f-a216-4acc-b4ff-e4ff1240fa92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:21.463267 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.463236    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"]
Jul 28 17:58:21.466725 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.466656    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79847955cc-vcl8v"]
Jul 28 17:58:21.498693 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.498647    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a49b43f-a216-4acc-b4ff-e4ff1240fa92" (UID: "6a49b43f-a216-4acc-b4ff-e4ff1240fa92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:21.533888 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.533855    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:21.533888 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:21.533884    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a49b43f-a216-4acc-b4ff-e4ff1240fa92-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:58:22.129863 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.129780    2585 generic.go:356] "Generic (PLEG): container finished" podID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerID="7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad" exitCode=0
Jul 28 17:58:22.129863 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.129850    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"
Jul 28 17:58:22.130398 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.129858    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" event={"ID":"6a49b43f-a216-4acc-b4ff-e4ff1240fa92","Type":"ContainerDied","Data":"7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad"}
Jul 28 17:58:22.130398 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.129947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x" event={"ID":"6a49b43f-a216-4acc-b4ff-e4ff1240fa92","Type":"ContainerDied","Data":"8b13e0a61d07831522e72fc76467eddf2cce3c3d09aebfeb22cd06c684058976"}
Jul 28 17:58:22.130398 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.129980    2585 scope.go:117] "RemoveContainer" containerID="7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad"
Jul 28 17:58:22.138657 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.138639    2585 scope.go:117] "RemoveContainer" containerID="eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec"
Jul 28 17:58:22.147125 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.147106    2585 scope.go:117] "RemoveContainer" containerID="7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad"
Jul 28 17:58:22.147380 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:58:22.147360    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad\": container with ID starting with 7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad not found: ID does not exist" containerID="7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad"
Jul 28 17:58:22.147434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.147389    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad"} err="failed to get container status \"7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad\": rpc error: code = NotFound desc = could not find container \"7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad\": container with ID starting with 7f53168126ff122ed8e023ea9af3c2dec34415106c9e052dc3e43a5ad255e6ad not found: ID does not exist"
Jul 28 17:58:22.147434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.147408    2585 scope.go:117] "RemoveContainer" containerID="eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec"
Jul 28 17:58:22.147652 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:58:22.147637    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec\": container with ID starting with eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec not found: ID does not exist" containerID="eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec"
Jul 28 17:58:22.147700 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.147657    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec"} err="failed to get container status \"eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec\": rpc error: code = NotFound desc = could not find container \"eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec\": container with ID starting with eee623f633dbab231ae48da2b5fb5d4a0dd1421873b452544b6f8cbe99a630ec not found: ID does not exist"
Jul 28 17:58:22.151147 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.151124    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"]
Jul 28 17:58:22.157719 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:22.157697    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6544489598-z5f7x"]
Jul 28 17:58:23.180117 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:23.180088    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" path="/var/lib/kubelet/pods/6a49b43f-a216-4acc-b4ff-e4ff1240fa92/volumes"
Jul 28 17:58:23.180517 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:23.180504    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" path="/var/lib/kubelet/pods/98a48a03-eb95-4bad-bec4-19c78386d55f/volumes"
Jul 28 17:58:44.721392 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721353    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"]
Jul 28 17:58:44.721882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721793    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerName="main"
Jul 28 17:58:44.721882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721812    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerName="main"
Jul 28 17:58:44.721882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721823    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="storage-initializer"
Jul 28 17:58:44.721882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721831    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="storage-initializer"
Jul 28 17:58:44.721882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721857    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerName="storage-initializer"
Jul 28 17:58:44.721882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721865    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerName="storage-initializer"
Jul 28 17:58:44.721882 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721881    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="main"
Jul 28 17:58:44.722228 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721890    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="main"
Jul 28 17:58:44.722228 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721968    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a48a03-eb95-4bad-bec4-19c78386d55f" containerName="main"
Jul 28 17:58:44.722228 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.721980    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a49b43f-a216-4acc-b4ff-e4ff1240fa92" containerName="main"
Jul 28 17:58:44.724195 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.724174    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.727979 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.727954    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-fxwf7\""
Jul 28 17:58:44.729079 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.729062    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 17:58:44.729319 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.729294    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 17:58:44.729388 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.729318    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 17:58:44.732190 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.732172    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 17:58:44.755801 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.755775    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"]
Jul 28 17:58:44.829434 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.829404    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7k5\" (UniqueName: \"kubernetes.io/projected/6ad95427-0690-41fe-bb8d-f1aece255e6c-kube-api-access-dw7k5\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.829579 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.829443    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-dshm\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.829579 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.829473    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.829669 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.829579    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-model-cache\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.829669 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.829623    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad95427-0690-41fe-bb8d-f1aece255e6c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.829669 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.829639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-home\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.829774 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.829677    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930382 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930346    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-model-cache\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930382 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930381    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad95427-0690-41fe-bb8d-f1aece255e6c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930653 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930403    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-home\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930653 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930443    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930653 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930462    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7k5\" (UniqueName: \"kubernetes.io/projected/6ad95427-0690-41fe-bb8d-f1aece255e6c-kube-api-access-dw7k5\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930653 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930498    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-dshm\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930653 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.930984 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930962    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-model-cache\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.931042 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.930996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.931042 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.931031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-home\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.931140 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.931056    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.932783 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.932756    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-dshm\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.932970 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.932954    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad95427-0690-41fe-bb8d-f1aece255e6c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:44.945208 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:44.945183    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7k5\" (UniqueName: \"kubernetes.io/projected/6ad95427-0690-41fe-bb8d-f1aece255e6c-kube-api-access-dw7k5\") pod \"tokenizer-migration-path-test-kserve-754b496dd-6bqc9\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:45.034700 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.034664    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:45.093004 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.092529    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"]
Jul 28 17:58:45.095359 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.095337    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.108624 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.108599    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"]
Jul 28 17:58:45.132881 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.132841    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.133003 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.132905    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1910dbc-881c-44da-8395-c127fe1243e4-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.133003 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.132954    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.188287 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.188264    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"]
Jul 28 17:58:45.189676 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:58:45.189653    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad95427_0690_41fe_bb8d_f1aece255e6c.slice/crio-f3b58de69d6a51fac4c7cb67890d9c6d92545395121fbe18155abc544d965710 WatchSource:0}: Error finding container f3b58de69d6a51fac4c7cb67890d9c6d92545395121fbe18155abc544d965710: Status 404 returned error can't find the container with id f3b58de69d6a51fac4c7cb67890d9c6d92545395121fbe18155abc544d965710
Jul 28 17:58:45.231843 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.231814    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" event={"ID":"6ad95427-0690-41fe-bb8d-f1aece255e6c","Type":"ContainerStarted","Data":"f3b58de69d6a51fac4c7cb67890d9c6d92545395121fbe18155abc544d965710"}
Jul 28 17:58:45.233521 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.233483    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1910dbc-881c-44da-8395-c127fe1243e4-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.233638 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.233536    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.233638 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.233627    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.233892 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.233867    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.233975 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.233933    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.235803 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.235783    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1910dbc-881c-44da-8395-c127fe1243e4-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.408440 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.408356    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:45.543342 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:45.543318    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"]
Jul 28 17:58:45.545382 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:58:45.545355    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1910dbc_881c_44da_8395_c127fe1243e4.slice/crio-cff1c8b5ea961d24ea7988c55f7d7cb95bb945dea0be4874298daf59761cc275 WatchSource:0}: Error finding container cff1c8b5ea961d24ea7988c55f7d7cb95bb945dea0be4874298daf59761cc275: Status 404 returned error can't find the container with id cff1c8b5ea961d24ea7988c55f7d7cb95bb945dea0be4874298daf59761cc275
Jul 28 17:58:46.240038 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:46.239999    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" event={"ID":"e1910dbc-881c-44da-8395-c127fe1243e4","Type":"ContainerStarted","Data":"15e3581950007a3f3d9aafcb3afca964ac8d06c1ff8581ee217b5f07d5e2cf59"}
Jul 28 17:58:46.240541 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:46.240067    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" event={"ID":"e1910dbc-881c-44da-8395-c127fe1243e4","Type":"ContainerStarted","Data":"cff1c8b5ea961d24ea7988c55f7d7cb95bb945dea0be4874298daf59761cc275"}
Jul 28 17:58:46.242617 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:46.242586    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" event={"ID":"6ad95427-0690-41fe-bb8d-f1aece255e6c","Type":"ContainerStarted","Data":"a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b"}
Jul 28 17:58:50.264569 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:50.264521    2585 generic.go:356] "Generic (PLEG): container finished" podID="e1910dbc-881c-44da-8395-c127fe1243e4" containerID="15e3581950007a3f3d9aafcb3afca964ac8d06c1ff8581ee217b5f07d5e2cf59" exitCode=0
Jul 28 17:58:50.265091 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:50.264594    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" event={"ID":"e1910dbc-881c-44da-8395-c127fe1243e4","Type":"ContainerDied","Data":"15e3581950007a3f3d9aafcb3afca964ac8d06c1ff8581ee217b5f07d5e2cf59"}
Jul 28 17:58:50.266196 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:50.266175    2585 generic.go:356] "Generic (PLEG): container finished" podID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerID="a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b" exitCode=0
Jul 28 17:58:50.266315 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:50.266242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" event={"ID":"6ad95427-0690-41fe-bb8d-f1aece255e6c","Type":"ContainerDied","Data":"a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b"}
Jul 28 17:58:51.272753 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:51.272711    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" event={"ID":"e1910dbc-881c-44da-8395-c127fe1243e4","Type":"ContainerStarted","Data":"4d9a864d859c490380ad6c53177b74e784f3bbf007bd3b507665fe282bc562ed"}
Jul 28 17:58:51.273208 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:51.272962    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:58:51.274665 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:51.274643    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" event={"ID":"6ad95427-0690-41fe-bb8d-f1aece255e6c","Type":"ContainerStarted","Data":"196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405"}
Jul 28 17:58:51.292476 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:51.292425    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" podStartSLOduration=2.068892229 podStartE2EDuration="6.292408055s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.223515826s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:51.291324596 +0000 UTC m=+2516.634916873" watchObservedRunningTime="2026-07-28 17:58:51.292408055 +0000 UTC m=+2516.636000371"
Jul 28 17:58:51.318609 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:51.318543    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" podStartSLOduration=2.809967981 podStartE2EDuration="7.318526292s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.508558311s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:51.31618695 +0000 UTC m=+2516.659779228" watchObservedRunningTime="2026-07-28 17:58:51.318526292 +0000 UTC m=+2516.662118572"
Jul 28 17:58:52.035547 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:52.035498    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:55.035194 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:55.035156    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:55.047740 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:55.047717    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:58:55.306003 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:58:55.305977    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:59:22.291456 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:22.291422    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:59:23.606377 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:23.606345    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd_e1910dbc-881c-44da-8395-c127fe1243e4/main/0.log"
Jul 28 17:59:23.840579 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:23.840534    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"]
Jul 28 17:59:23.849275 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:23.849244    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"]
Jul 28 17:59:23.849501 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:23.849470    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" podUID="e1910dbc-881c-44da-8395-c127fe1243e4" containerName="main" containerID="cri-o://4d9a864d859c490380ad6c53177b74e784f3bbf007bd3b507665fe282bc562ed" gracePeriod=30
Jul 28 17:59:23.850333 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:59:23.850300    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:59:23.850333 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:23.850333 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:23.850333 ip-10-0-135-20 kubenswrapper[2585]:  > containerID="196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405" cmd=["/bin/sleep","15"]
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:23.850360    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" message="" err=<
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]:  >
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:59:23.850377    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerName="main" containerID="cri-o://196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405"
Jul 28 17:59:23.850503 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:23.850425    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerName="main" containerID="cri-o://196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405" gracePeriod=60
Jul 28 17:59:24.193566 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.193530    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:59:24.281616 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281577    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-tmp-dir\") pod \"6ad95427-0690-41fe-bb8d-f1aece255e6c\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") "
Jul 28 17:59:24.281804 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281627    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-dshm\") pod \"6ad95427-0690-41fe-bb8d-f1aece255e6c\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") "
Jul 28 17:59:24.281804 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281646    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad95427-0690-41fe-bb8d-f1aece255e6c-tls-certs\") pod \"6ad95427-0690-41fe-bb8d-f1aece255e6c\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") "
Jul 28 17:59:24.281804 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281677    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-home\") pod \"6ad95427-0690-41fe-bb8d-f1aece255e6c\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") "
Jul 28 17:59:24.281804 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281705    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw7k5\" (UniqueName: \"kubernetes.io/projected/6ad95427-0690-41fe-bb8d-f1aece255e6c-kube-api-access-dw7k5\") pod \"6ad95427-0690-41fe-bb8d-f1aece255e6c\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") "
Jul 28 17:59:24.281804 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281740    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-kserve-provision-location\") pod \"6ad95427-0690-41fe-bb8d-f1aece255e6c\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") "
Jul 28 17:59:24.281804 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281765    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-model-cache\") pod \"6ad95427-0690-41fe-bb8d-f1aece255e6c\" (UID: \"6ad95427-0690-41fe-bb8d-f1aece255e6c\") "
Jul 28 17:59:24.282085 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281894    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ad95427-0690-41fe-bb8d-f1aece255e6c" (UID: "6ad95427-0690-41fe-bb8d-f1aece255e6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:24.282085 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.281981    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-home" (OuterVolumeSpecName: "home") pod "6ad95427-0690-41fe-bb8d-f1aece255e6c" (UID: "6ad95427-0690-41fe-bb8d-f1aece255e6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:24.282192 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.282096    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.282192 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.282107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ad95427-0690-41fe-bb8d-f1aece255e6c" (UID: "6ad95427-0690-41fe-bb8d-f1aece255e6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:24.282192 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.282115    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.283993 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.283913    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad95427-0690-41fe-bb8d-f1aece255e6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ad95427-0690-41fe-bb8d-f1aece255e6c" (UID: "6ad95427-0690-41fe-bb8d-f1aece255e6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:24.284126 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.284022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad95427-0690-41fe-bb8d-f1aece255e6c-kube-api-access-dw7k5" (OuterVolumeSpecName: "kube-api-access-dw7k5") pod "6ad95427-0690-41fe-bb8d-f1aece255e6c" (UID: "6ad95427-0690-41fe-bb8d-f1aece255e6c"). InnerVolumeSpecName "kube-api-access-dw7k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:24.284415 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.284399    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-dshm" (OuterVolumeSpecName: "dshm") pod "6ad95427-0690-41fe-bb8d-f1aece255e6c" (UID: "6ad95427-0690-41fe-bb8d-f1aece255e6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:24.341410 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.341374    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ad95427-0690-41fe-bb8d-f1aece255e6c" (UID: "6ad95427-0690-41fe-bb8d-f1aece255e6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:24.382545 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.382512    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw7k5\" (UniqueName: \"kubernetes.io/projected/6ad95427-0690-41fe-bb8d-f1aece255e6c-kube-api-access-dw7k5\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.382545 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.382540    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.382545 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.382565    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.382825 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.382577    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad95427-0690-41fe-bb8d-f1aece255e6c-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.382825 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.382591    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad95427-0690-41fe-bb8d-f1aece255e6c-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.425401 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.425374    2585 generic.go:356] "Generic (PLEG): container finished" podID="e1910dbc-881c-44da-8395-c127fe1243e4" containerID="4d9a864d859c490380ad6c53177b74e784f3bbf007bd3b507665fe282bc562ed" exitCode=0
Jul 28 17:59:24.425540 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.425455    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" event={"ID":"e1910dbc-881c-44da-8395-c127fe1243e4","Type":"ContainerDied","Data":"4d9a864d859c490380ad6c53177b74e784f3bbf007bd3b507665fe282bc562ed"}
Jul 28 17:59:24.427194 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.427174    2585 generic.go:356] "Generic (PLEG): container finished" podID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerID="196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405" exitCode=0
Jul 28 17:59:24.427318 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.427206    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" event={"ID":"6ad95427-0690-41fe-bb8d-f1aece255e6c","Type":"ContainerDied","Data":"196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405"}
Jul 28 17:59:24.427318 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.427241    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"
Jul 28 17:59:24.427318 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.427251    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9" event={"ID":"6ad95427-0690-41fe-bb8d-f1aece255e6c","Type":"ContainerDied","Data":"f3b58de69d6a51fac4c7cb67890d9c6d92545395121fbe18155abc544d965710"}
Jul 28 17:59:24.427318 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.427281    2585 scope.go:117] "RemoveContainer" containerID="196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405"
Jul 28 17:59:24.438345 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.438321    2585 scope.go:117] "RemoveContainer" containerID="a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b"
Jul 28 17:59:24.447335 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.447314    2585 scope.go:117] "RemoveContainer" containerID="196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405"
Jul 28 17:59:24.447644 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:59:24.447624    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405\": container with ID starting with 196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405 not found: ID does not exist" containerID="196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405"
Jul 28 17:59:24.447698 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.447654    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405"} err="failed to get container status \"196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405\": rpc error: code = NotFound desc = could not find container \"196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405\": container with ID starting with 196568466dc9dbe12e3a72f991d745543b46d1caeb69e4076110ae95340a2405 not found: ID does not exist"
Jul 28 17:59:24.447698 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.447674    2585 scope.go:117] "RemoveContainer" containerID="a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b"
Jul 28 17:59:24.447938 ip-10-0-135-20 kubenswrapper[2585]: E0728 17:59:24.447924    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b\": container with ID starting with a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b not found: ID does not exist" containerID="a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b"
Jul 28 17:59:24.447983 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.447942    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b"} err="failed to get container status \"a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b\": rpc error: code = NotFound desc = could not find container \"a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b\": container with ID starting with a3e000358dd532454fb5353e3663097dc5ac8c64c6ba779fef9237f7bae8788b not found: ID does not exist"
Jul 28 17:59:24.455125 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.455102    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"]
Jul 28 17:59:24.459915 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.459890    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-754b496dd-6bqc9"]
Jul 28 17:59:24.489566 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.489534    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:59:24.583665 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.583573    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1910dbc-881c-44da-8395-c127fe1243e4-tls-certs\") pod \"e1910dbc-881c-44da-8395-c127fe1243e4\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") "
Jul 28 17:59:24.583812 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.583683    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-kserve-provision-location\") pod \"e1910dbc-881c-44da-8395-c127fe1243e4\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") "
Jul 28 17:59:24.583812 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.583730    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-tmp-dir\") pod \"e1910dbc-881c-44da-8395-c127fe1243e4\" (UID: \"e1910dbc-881c-44da-8395-c127fe1243e4\") "
Jul 28 17:59:24.584084 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.584050    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1910dbc-881c-44da-8395-c127fe1243e4" (UID: "e1910dbc-881c-44da-8395-c127fe1243e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:24.585508 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.585489    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1910dbc-881c-44da-8395-c127fe1243e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1910dbc-881c-44da-8395-c127fe1243e4" (UID: "e1910dbc-881c-44da-8395-c127fe1243e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:24.640736 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.640705    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1910dbc-881c-44da-8395-c127fe1243e4" (UID: "e1910dbc-881c-44da-8395-c127fe1243e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:24.684985 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.684956    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.684985 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.684982    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1910dbc-881c-44da-8395-c127fe1243e4-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:24.685200 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:24.684992    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1910dbc-881c-44da-8395-c127fe1243e4-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 17:59:25.180300 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:25.180262    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" path="/var/lib/kubelet/pods/6ad95427-0690-41fe-bb8d-f1aece255e6c/volumes"
Jul 28 17:59:25.432823 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:25.432797    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"
Jul 28 17:59:25.433023 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:25.432797    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd" event={"ID":"e1910dbc-881c-44da-8395-c127fe1243e4","Type":"ContainerDied","Data":"cff1c8b5ea961d24ea7988c55f7d7cb95bb945dea0be4874298daf59761cc275"}
Jul 28 17:59:25.433023 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:25.432945    2585 scope.go:117] "RemoveContainer" containerID="4d9a864d859c490380ad6c53177b74e784f3bbf007bd3b507665fe282bc562ed"
Jul 28 17:59:25.441058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:25.441014    2585 scope.go:117] "RemoveContainer" containerID="15e3581950007a3f3d9aafcb3afca964ac8d06c1ff8581ee217b5f07d5e2cf59"
Jul 28 17:59:25.450414 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:25.450369    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"]
Jul 28 17:59:25.451769 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:25.451750    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-54f48b8b68-kp6bd"]
Jul 28 17:59:27.181058 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:27.181020    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1910dbc-881c-44da-8395-c127fe1243e4" path="/var/lib/kubelet/pods/e1910dbc-881c-44da-8395-c127fe1243e4/volumes"
Jul 28 17:59:52.419699 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.419661    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"]
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420081    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1910dbc-881c-44da-8395-c127fe1243e4" containerName="main"
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420099    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1910dbc-881c-44da-8395-c127fe1243e4" containerName="main"
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420128    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1910dbc-881c-44da-8395-c127fe1243e4" containerName="storage-initializer"
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420136    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1910dbc-881c-44da-8395-c127fe1243e4" containerName="storage-initializer"
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420159    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerName="main"
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420167    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerName="main"
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420178    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerName="storage-initializer"
Jul 28 17:59:52.420237 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420187    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerName="storage-initializer"
Jul 28 17:59:52.420673 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420267    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1910dbc-881c-44da-8395-c127fe1243e4" containerName="main"
Jul 28 17:59:52.420673 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.420281    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad95427-0690-41fe-bb8d-f1aece255e6c" containerName="main"
Jul 28 17:59:52.422373 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.422352    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.426745 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.426565    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 17:59:52.426745 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.426641    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 17:59:52.426745 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.426630    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 17:59:52.426983 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.426757    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-f8g4c\""
Jul 28 17:59:52.426983 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.426967    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-pwpmm\""
Jul 28 17:59:52.433962 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.433939    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"]
Jul 28 17:59:52.505637 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.505611    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6ks\" (UniqueName: \"kubernetes.io/projected/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-kube-api-access-wh6ks\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.505773 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.505676    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.606708 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.606673    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.606880 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.606735    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6ks\" (UniqueName: \"kubernetes.io/projected/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-kube-api-access-wh6ks\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.609131 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.609111    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.616032 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.616005    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6ks\" (UniqueName: \"kubernetes.io/projected/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-kube-api-access-wh6ks\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.733244 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.733176    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:52.865923 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:52.865898    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"]
Jul 28 17:59:52.868367 ip-10-0-135-20 kubenswrapper[2585]: W0728 17:59:52.868344    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4860c8b7_e5aa_424e_aa20_b03cc793c9b9.slice/crio-fe83cb2018b2d45820d3c9400958828468b28321659faa35a51ffe07ccfe10fa WatchSource:0}: Error finding container fe83cb2018b2d45820d3c9400958828468b28321659faa35a51ffe07ccfe10fa: Status 404 returned error can't find the container with id fe83cb2018b2d45820d3c9400958828468b28321659faa35a51ffe07ccfe10fa
Jul 28 17:59:53.547755 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:53.547722    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh" event={"ID":"4860c8b7-e5aa-424e-aa20-b03cc793c9b9","Type":"ContainerStarted","Data":"6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa"}
Jul 28 17:59:53.548149 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:53.547779    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh" event={"ID":"4860c8b7-e5aa-424e-aa20-b03cc793c9b9","Type":"ContainerStarted","Data":"fe83cb2018b2d45820d3c9400958828468b28321659faa35a51ffe07ccfe10fa"}
Jul 28 17:59:53.548149 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:53.547809    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 17:59:53.572046 ip-10-0-135-20 kubenswrapper[2585]: I0728 17:59:53.572000    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh" podStartSLOduration=1.571985591 podStartE2EDuration="1.571985591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:53.570978436 +0000 UTC m=+2578.914570713" watchObservedRunningTime="2026-07-28 17:59:53.571985591 +0000 UTC m=+2578.915577886"
Jul 28 18:00:24.553818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:24.553742    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 18:00:25.930689 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:25.930652    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"]
Jul 28 18:00:34.554357 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:34.554310    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh" podUID="4860c8b7-e5aa-424e-aa20-b03cc793c9b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:00:40.949685 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:40.949644    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh" podUID="4860c8b7-e5aa-424e-aa20-b03cc793c9b9" containerName="main" containerID="cri-o://6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa" gracePeriod=45
Jul 28 18:00:41.193221 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.193196    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 18:00:41.308458 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.308428    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh6ks\" (UniqueName: \"kubernetes.io/projected/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-kube-api-access-wh6ks\") pod \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") "
Jul 28 18:00:41.308644 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.308472    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-tls-certs\") pod \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\" (UID: \"4860c8b7-e5aa-424e-aa20-b03cc793c9b9\") "
Jul 28 18:00:41.310608 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.310577    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4860c8b7-e5aa-424e-aa20-b03cc793c9b9" (UID: "4860c8b7-e5aa-424e-aa20-b03cc793c9b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:41.310744 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.310654    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-kube-api-access-wh6ks" (OuterVolumeSpecName: "kube-api-access-wh6ks") pod "4860c8b7-e5aa-424e-aa20-b03cc793c9b9" (UID: "4860c8b7-e5aa-424e-aa20-b03cc793c9b9"). InnerVolumeSpecName "kube-api-access-wh6ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:41.409799 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.409771    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6ks\" (UniqueName: \"kubernetes.io/projected/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-kube-api-access-wh6ks\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:00:41.409799 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.409796    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4860c8b7-e5aa-424e-aa20-b03cc793c9b9-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:00:41.744278 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.744190    2585 generic.go:356] "Generic (PLEG): container finished" podID="4860c8b7-e5aa-424e-aa20-b03cc793c9b9" containerID="6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa" exitCode=0
Jul 28 18:00:41.744278 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.744245    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"
Jul 28 18:00:41.744278 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.744258    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh" event={"ID":"4860c8b7-e5aa-424e-aa20-b03cc793c9b9","Type":"ContainerDied","Data":"6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa"}
Jul 28 18:00:41.744491 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.744311    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh" event={"ID":"4860c8b7-e5aa-424e-aa20-b03cc793c9b9","Type":"ContainerDied","Data":"fe83cb2018b2d45820d3c9400958828468b28321659faa35a51ffe07ccfe10fa"}
Jul 28 18:00:41.744491 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.744344    2585 scope.go:117] "RemoveContainer" containerID="6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa"
Jul 28 18:00:41.753109 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.753087    2585 scope.go:117] "RemoveContainer" containerID="6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa"
Jul 28 18:00:41.753350 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:00:41.753331    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa\": container with ID starting with 6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa not found: ID does not exist" containerID="6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa"
Jul 28 18:00:41.753419 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.753355    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa"} err="failed to get container status \"6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa\": rpc error: code = NotFound desc = could not find container \"6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa\": container with ID starting with 6317142b40b7b9d8d360fd0e6429abcfcae2b3095bfdeb554c4e9ecde394edaa not found: ID does not exist"
Jul 28 18:00:41.770624 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.770598    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"]
Jul 28 18:00:41.783563 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:41.783527    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefdjfh"]
Jul 28 18:00:43.181064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:43.181029    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4860c8b7-e5aa-424e-aa20-b03cc793c9b9" path="/var/lib/kubelet/pods/4860c8b7-e5aa-424e-aa20-b03cc793c9b9/volumes"
Jul 28 18:00:51.425519 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.425487    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"]
Jul 28 18:00:51.426008 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.425975    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4860c8b7-e5aa-424e-aa20-b03cc793c9b9" containerName="main"
Jul 28 18:00:51.426008 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.425995    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4860c8b7-e5aa-424e-aa20-b03cc793c9b9" containerName="main"
Jul 28 18:00:51.426125 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.426075    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4860c8b7-e5aa-424e-aa20-b03cc793c9b9" containerName="main"
Jul 28 18:00:51.428815 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.428795    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.431999 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.431963    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 18:00:51.432105 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.432024    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 18:00:51.433215 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.433190    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-rw9xd\""
Jul 28 18:00:51.433324 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.433231    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 18:00:51.433324 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.433313    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-n26hm\""
Jul 28 18:00:51.466587 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.466565    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"]
Jul 28 18:00:51.496040 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.495995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bb998f-091e-4265-a940-b16474beedcc-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.496174 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.496054    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84pj\" (UniqueName: \"kubernetes.io/projected/16bb998f-091e-4265-a940-b16474beedcc-kube-api-access-k84pj\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.596979 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.596952    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84pj\" (UniqueName: \"kubernetes.io/projected/16bb998f-091e-4265-a940-b16474beedcc-kube-api-access-k84pj\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.597137 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.597060    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bb998f-091e-4265-a940-b16474beedcc-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.599490 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.599460    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bb998f-091e-4265-a940-b16474beedcc-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.611752 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.611730    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84pj\" (UniqueName: \"kubernetes.io/projected/16bb998f-091e-4265-a940-b16474beedcc-kube-api-access-k84pj\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.738479 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.738385    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:51.872896 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:51.872861    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"]
Jul 28 18:00:51.874917 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:00:51.874887    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bb998f_091e_4265_a940_b16474beedcc.slice/crio-df90dc56562a16e803cd46befed61073fd2ca5979b1476786dff94e1a125a189 WatchSource:0}: Error finding container df90dc56562a16e803cd46befed61073fd2ca5979b1476786dff94e1a125a189: Status 404 returned error can't find the container with id df90dc56562a16e803cd46befed61073fd2ca5979b1476786dff94e1a125a189
Jul 28 18:00:52.793694 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:52.793658    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb" event={"ID":"16bb998f-091e-4265-a940-b16474beedcc","Type":"ContainerStarted","Data":"bb3c4fc75e8d20cd5f1ffb5b22a74c388782cf5c1bad497ccf728c18046fea5b"}
Jul 28 18:00:52.794161 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:52.793712    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb" event={"ID":"16bb998f-091e-4265-a940-b16474beedcc","Type":"ContainerStarted","Data":"df90dc56562a16e803cd46befed61073fd2ca5979b1476786dff94e1a125a189"}
Jul 28 18:00:52.794161 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:52.793735    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:00:52.818207 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:00:52.818161    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb" podStartSLOduration=1.818147543 podStartE2EDuration="1.818147543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:52.817513749 +0000 UTC m=+2638.161106109" watchObservedRunningTime="2026-07-28 18:00:52.818147543 +0000 UTC m=+2638.161739828"
Jul 28 18:01:23.800499 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:23.800463    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:01:50.929336 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:50.929295    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"]
Jul 28 18:01:50.931941 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:50.931918    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:50.936196 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:50.936164    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 18:01:50.937493 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:50.937473    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-xh9hx\""
Jul 28 18:01:50.943681 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:50.943659    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"]
Jul 28 18:01:51.097135 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.097095    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff86fa-d05f-4df0-966c-959df53364f9-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:51.097312 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.097160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/3fff86fa-d05f-4df0-966c-959df53364f9-kube-api-access-j8dqm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:51.198469 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.198385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff86fa-d05f-4df0-966c-959df53364f9-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:51.198469 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.198438    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/3fff86fa-d05f-4df0-966c-959df53364f9-kube-api-access-j8dqm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:51.200855 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.200821    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff86fa-d05f-4df0-966c-959df53364f9-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:51.207800 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.207774    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/3fff86fa-d05f-4df0-966c-959df53364f9-kube-api-access-j8dqm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:51.241354 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.241329    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:51.383928 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:51.383895    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"]
Jul 28 18:01:51.384603 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:01:51.384540    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fff86fa_d05f_4df0_966c_959df53364f9.slice/crio-cc866b50b2f2fd3da12b31ab9122eb216aeb95f3cdae5c831bea009f5074444f WatchSource:0}: Error finding container cc866b50b2f2fd3da12b31ab9122eb216aeb95f3cdae5c831bea009f5074444f: Status 404 returned error can't find the container with id cc866b50b2f2fd3da12b31ab9122eb216aeb95f3cdae5c831bea009f5074444f
Jul 28 18:01:52.041396 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:52.041358    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9" event={"ID":"3fff86fa-d05f-4df0-966c-959df53364f9","Type":"ContainerStarted","Data":"0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c"}
Jul 28 18:01:52.041808 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:52.041418    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9" event={"ID":"3fff86fa-d05f-4df0-966c-959df53364f9","Type":"ContainerStarted","Data":"cc866b50b2f2fd3da12b31ab9122eb216aeb95f3cdae5c831bea009f5074444f"}
Jul 28 18:01:52.041808 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:52.041445    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:01:52.062617 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:52.062533    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9" podStartSLOduration=2.062518185 podStartE2EDuration="2.062518185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:52.060579826 +0000 UTC m=+2697.404172100" watchObservedRunningTime="2026-07-28 18:01:52.062518185 +0000 UTC m=+2697.406110462"
Jul 28 18:01:55.215969 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:55.215851    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:01:55.221402 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:01:55.221383    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:02:23.047421 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:23.047391    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:02:24.146134 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:24.146098    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"]
Jul 28 18:02:33.047684 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:33.047638    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9" podUID="3fff86fa-d05f-4df0-966c-959df53364f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:02:39.165896 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.165837    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9" podUID="3fff86fa-d05f-4df0-966c-959df53364f9" containerName="main" containerID="cri-o://0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c" gracePeriod=45
Jul 28 18:02:39.411621 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.411597    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:02:39.503344 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.503256    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/3fff86fa-d05f-4df0-966c-959df53364f9-kube-api-access-j8dqm\") pod \"3fff86fa-d05f-4df0-966c-959df53364f9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") "
Jul 28 18:02:39.503344 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.503333    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff86fa-d05f-4df0-966c-959df53364f9-tls-certs\") pod \"3fff86fa-d05f-4df0-966c-959df53364f9\" (UID: \"3fff86fa-d05f-4df0-966c-959df53364f9\") "
Jul 28 18:02:39.505436 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.505407    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fff86fa-d05f-4df0-966c-959df53364f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fff86fa-d05f-4df0-966c-959df53364f9" (UID: "3fff86fa-d05f-4df0-966c-959df53364f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:39.505436 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.505405    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fff86fa-d05f-4df0-966c-959df53364f9-kube-api-access-j8dqm" (OuterVolumeSpecName: "kube-api-access-j8dqm") pod "3fff86fa-d05f-4df0-966c-959df53364f9" (UID: "3fff86fa-d05f-4df0-966c-959df53364f9"). InnerVolumeSpecName "kube-api-access-j8dqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:39.603913 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.603882    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff86fa-d05f-4df0-966c-959df53364f9-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:02:39.603913 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:39.603910    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/3fff86fa-d05f-4df0-966c-959df53364f9-kube-api-access-j8dqm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:02:40.241579 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.241529    2585 generic.go:356] "Generic (PLEG): container finished" podID="3fff86fa-d05f-4df0-966c-959df53364f9" containerID="0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c" exitCode=0
Jul 28 18:02:40.241579 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.241587    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"
Jul 28 18:02:40.242064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.241606    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9" event={"ID":"3fff86fa-d05f-4df0-966c-959df53364f9","Type":"ContainerDied","Data":"0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c"}
Jul 28 18:02:40.242064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.241647    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9" event={"ID":"3fff86fa-d05f-4df0-966c-959df53364f9","Type":"ContainerDied","Data":"cc866b50b2f2fd3da12b31ab9122eb216aeb95f3cdae5c831bea009f5074444f"}
Jul 28 18:02:40.242064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.241668    2585 scope.go:117] "RemoveContainer" containerID="0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c"
Jul 28 18:02:40.250064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.249900    2585 scope.go:117] "RemoveContainer" containerID="0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c"
Jul 28 18:02:40.250158 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:02:40.250138    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c\": container with ID starting with 0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c not found: ID does not exist" containerID="0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c"
Jul 28 18:02:40.250225 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.250173    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c"} err="failed to get container status \"0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c\": rpc error: code = NotFound desc = could not find container \"0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c\": container with ID starting with 0049c710c3c61c2c5531c24c82c8186439059a78464fd86f0422abf2468bc57c not found: ID does not exist"
Jul 28 18:02:40.264667 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.264643    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"]
Jul 28 18:02:40.268508 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:40.268488    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen4lw9"]
Jul 28 18:02:41.180793 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:41.180760    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fff86fa-d05f-4df0-966c-959df53364f9" path="/var/lib/kubelet/pods/3fff86fa-d05f-4df0-966c-959df53364f9/volumes"
Jul 28 18:02:44.075396 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:44.075362    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"]
Jul 28 18:02:53.799767 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:53.799731    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb" podUID="16bb998f-091e-4265-a940-b16474beedcc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:02:59.094067 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.094018    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb" podUID="16bb998f-091e-4265-a940-b16474beedcc" containerName="main" containerID="cri-o://bb3c4fc75e8d20cd5f1ffb5b22a74c388782cf5c1bad497ccf728c18046fea5b" gracePeriod=45
Jul 28 18:02:59.320345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.320319    2585 generic.go:356] "Generic (PLEG): container finished" podID="16bb998f-091e-4265-a940-b16474beedcc" containerID="bb3c4fc75e8d20cd5f1ffb5b22a74c388782cf5c1bad497ccf728c18046fea5b" exitCode=0
Jul 28 18:02:59.320421 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.320394    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb" event={"ID":"16bb998f-091e-4265-a940-b16474beedcc","Type":"ContainerDied","Data":"bb3c4fc75e8d20cd5f1ffb5b22a74c388782cf5c1bad497ccf728c18046fea5b"}
Jul 28 18:02:59.320477 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.320460    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb" event={"ID":"16bb998f-091e-4265-a940-b16474beedcc","Type":"ContainerDied","Data":"df90dc56562a16e803cd46befed61073fd2ca5979b1476786dff94e1a125a189"}
Jul 28 18:02:59.320518 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.320491    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df90dc56562a16e803cd46befed61073fd2ca5979b1476786dff94e1a125a189"
Jul 28 18:02:59.333524 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.333503    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:02:59.463493 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.463395    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bb998f-091e-4265-a940-b16474beedcc-tls-certs\") pod \"16bb998f-091e-4265-a940-b16474beedcc\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") "
Jul 28 18:02:59.463684 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.463513    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k84pj\" (UniqueName: \"kubernetes.io/projected/16bb998f-091e-4265-a940-b16474beedcc-kube-api-access-k84pj\") pod \"16bb998f-091e-4265-a940-b16474beedcc\" (UID: \"16bb998f-091e-4265-a940-b16474beedcc\") "
Jul 28 18:02:59.465528 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.465501    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bb998f-091e-4265-a940-b16474beedcc-kube-api-access-k84pj" (OuterVolumeSpecName: "kube-api-access-k84pj") pod "16bb998f-091e-4265-a940-b16474beedcc" (UID: "16bb998f-091e-4265-a940-b16474beedcc"). InnerVolumeSpecName "kube-api-access-k84pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:59.465640 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.465605    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bb998f-091e-4265-a940-b16474beedcc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16bb998f-091e-4265-a940-b16474beedcc" (UID: "16bb998f-091e-4265-a940-b16474beedcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:59.564942 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.564893    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bb998f-091e-4265-a940-b16474beedcc-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:02:59.564942 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:02:59.564937    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84pj\" (UniqueName: \"kubernetes.io/projected/16bb998f-091e-4265-a940-b16474beedcc-kube-api-access-k84pj\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.324321 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:00.324288    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"
Jul 28 18:03:00.346385 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:00.346356    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"]
Jul 28 18:03:00.350543 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:00.350520    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q5rb"]
Jul 28 18:03:01.180140 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:01.180097    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16bb998f-091e-4265-a940-b16474beedcc" path="/var/lib/kubelet/pods/16bb998f-091e-4265-a940-b16474beedcc/volumes"
Jul 28 18:03:53.803381 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.803343    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"]
Jul 28 18:03:53.803872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.803685    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fff86fa-d05f-4df0-966c-959df53364f9" containerName="main"
Jul 28 18:03:53.803872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.803698    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fff86fa-d05f-4df0-966c-959df53364f9" containerName="main"
Jul 28 18:03:53.803872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.803707    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bb998f-091e-4265-a940-b16474beedcc" containerName="main"
Jul 28 18:03:53.803872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.803717    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bb998f-091e-4265-a940-b16474beedcc" containerName="main"
Jul 28 18:03:53.803872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.803775    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fff86fa-d05f-4df0-966c-959df53364f9" containerName="main"
Jul 28 18:03:53.803872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.803785    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bb998f-091e-4265-a940-b16474beedcc" containerName="main"
Jul 28 18:03:53.806621 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.806601    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:53.809253 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.809229    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 18:03:53.810483 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.810464    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-sbsdx\""
Jul 28 18:03:53.810646 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.810515    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 18:03:53.810646 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.810514    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 18:03:53.816876 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.816854    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"]
Jul 28 18:03:53.922848 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.922814    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:53.923032 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.922871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e565a65-2e13-4acf-8a05-c2a51bd97425-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:53.923032 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.922929    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:53.923032 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.922980    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9s9\" (UniqueName: \"kubernetes.io/projected/4e565a65-2e13-4acf-8a05-c2a51bd97425-kube-api-access-8c9s9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:53.923032 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.923012    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:53.923211 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:53.923063    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.023747 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.023708    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.023747 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.023749    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.024003 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.023787    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.024003 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.023845    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e565a65-2e13-4acf-8a05-c2a51bd97425-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.024003 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.023876    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.024003 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.023917    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9s9\" (UniqueName: \"kubernetes.io/projected/4e565a65-2e13-4acf-8a05-c2a51bd97425-kube-api-access-8c9s9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.024277 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.024244    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.024406 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.024293    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.024406 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.024355    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.026121 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.026100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.026400 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.026384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e565a65-2e13-4acf-8a05-c2a51bd97425-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.032204 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.032180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9s9\" (UniqueName: \"kubernetes.io/projected/4e565a65-2e13-4acf-8a05-c2a51bd97425-kube-api-access-8c9s9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.117805 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.117727    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:03:54.243940 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.243911    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"]
Jul 28 18:03:54.246750 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:03:54.246723    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e565a65_2e13_4acf_8a05_c2a51bd97425.slice/crio-3a8cd3c9f52f8ab3a995e448c35eb67c594784224e3c60a54025540c359c8da0 WatchSource:0}: Error finding container 3a8cd3c9f52f8ab3a995e448c35eb67c594784224e3c60a54025540c359c8da0: Status 404 returned error can't find the container with id 3a8cd3c9f52f8ab3a995e448c35eb67c594784224e3c60a54025540c359c8da0
Jul 28 18:03:54.543797 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.543749    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" event={"ID":"4e565a65-2e13-4acf-8a05-c2a51bd97425","Type":"ContainerStarted","Data":"93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b"}
Jul 28 18:03:54.543981 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.543826    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" event={"ID":"4e565a65-2e13-4acf-8a05-c2a51bd97425","Type":"ContainerStarted","Data":"3a8cd3c9f52f8ab3a995e448c35eb67c594784224e3c60a54025540c359c8da0"}
Jul 28 18:03:54.563757 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:54.563705    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" podStartSLOduration=1.5636914320000002 podStartE2EDuration="1.563691432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:54.562316227 +0000 UTC m=+2819.905908505" watchObservedRunningTime="2026-07-28 18:03:54.563691432 +0000 UTC m=+2819.907283709"
Jul 28 18:03:55.117998 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:03:55.117964    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:04:04.118706 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:04.118666    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:04:04.131058 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:04.131030    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:04:04.599417 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:04.599387    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:04:28.255315 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.255275    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"]
Jul 28 18:04:28.265434 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:04:28.265399    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:04:28.265434 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:04:28.265434 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 18:04:28.265434 ip-10-0-135-20 kubenswrapper[2585]:  > containerID="93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b" cmd=["/bin/sleep","15"]
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.265449    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" message="" err=<
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]:  >
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:04:28.265462    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" podUID="4e565a65-2e13-4acf-8a05-c2a51bd97425" containerName="main" containerID="cri-o://93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b"
Jul 28 18:04:28.265712 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.265495    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" podUID="4e565a65-2e13-4acf-8a05-c2a51bd97425" containerName="main" containerID="cri-o://93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b" gracePeriod=60
Jul 28 18:04:28.619914 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.619885    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:04:28.688244 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.688212    2585 generic.go:356] "Generic (PLEG): container finished" podID="4e565a65-2e13-4acf-8a05-c2a51bd97425" containerID="93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b" exitCode=0
Jul 28 18:04:28.688407 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.688253    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"
Jul 28 18:04:28.688407 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.688262    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" event={"ID":"4e565a65-2e13-4acf-8a05-c2a51bd97425","Type":"ContainerDied","Data":"93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b"}
Jul 28 18:04:28.688407 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.688309    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" event={"ID":"4e565a65-2e13-4acf-8a05-c2a51bd97425","Type":"ContainerDied","Data":"3a8cd3c9f52f8ab3a995e448c35eb67c594784224e3c60a54025540c359c8da0"}
Jul 28 18:04:28.688407 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.688331    2585 scope.go:117] "RemoveContainer" containerID="93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b"
Jul 28 18:04:28.698076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.697386    2585 scope.go:117] "RemoveContainer" containerID="93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b"
Jul 28 18:04:28.698076 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:04:28.698014    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b\": container with ID starting with 93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b not found: ID does not exist" containerID="93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b"
Jul 28 18:04:28.698076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.698051    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b"} err="failed to get container status \"93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b\": rpc error: code = NotFound desc = could not find container \"93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b\": container with ID starting with 93ffba69414b21e7fc4141b3460e2c8643eead3e56fae77ace166c5a3c80181b not found: ID does not exist"
Jul 28 18:04:28.708428 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708402    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-dshm\") pod \"4e565a65-2e13-4acf-8a05-c2a51bd97425\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") "
Jul 28 18:04:28.708507 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708439    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-model-cache\") pod \"4e565a65-2e13-4acf-8a05-c2a51bd97425\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") "
Jul 28 18:04:28.708507 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708472    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-tmp-dir\") pod \"4e565a65-2e13-4acf-8a05-c2a51bd97425\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") "
Jul 28 18:04:28.708627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708528    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-home\") pod \"4e565a65-2e13-4acf-8a05-c2a51bd97425\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") "
Jul 28 18:04:28.708627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708609    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e565a65-2e13-4acf-8a05-c2a51bd97425-tls-certs\") pod \"4e565a65-2e13-4acf-8a05-c2a51bd97425\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") "
Jul 28 18:04:28.708739 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708650    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c9s9\" (UniqueName: \"kubernetes.io/projected/4e565a65-2e13-4acf-8a05-c2a51bd97425-kube-api-access-8c9s9\") pod \"4e565a65-2e13-4acf-8a05-c2a51bd97425\" (UID: \"4e565a65-2e13-4acf-8a05-c2a51bd97425\") "
Jul 28 18:04:28.708794 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708707    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e565a65-2e13-4acf-8a05-c2a51bd97425" (UID: "4e565a65-2e13-4acf-8a05-c2a51bd97425"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:28.708794 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708776    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-home" (OuterVolumeSpecName: "home") pod "4e565a65-2e13-4acf-8a05-c2a51bd97425" (UID: "4e565a65-2e13-4acf-8a05-c2a51bd97425"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:28.708892 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708793    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e565a65-2e13-4acf-8a05-c2a51bd97425" (UID: "4e565a65-2e13-4acf-8a05-c2a51bd97425"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:28.708961 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708935    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:04:28.708961 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708955    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:04:28.709035 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.708968    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:04:28.710569 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.710534    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-dshm" (OuterVolumeSpecName: "dshm") pod "4e565a65-2e13-4acf-8a05-c2a51bd97425" (UID: "4e565a65-2e13-4acf-8a05-c2a51bd97425"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:28.710712 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.710688    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e565a65-2e13-4acf-8a05-c2a51bd97425-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e565a65-2e13-4acf-8a05-c2a51bd97425" (UID: "4e565a65-2e13-4acf-8a05-c2a51bd97425"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:28.710779 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.710744    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e565a65-2e13-4acf-8a05-c2a51bd97425-kube-api-access-8c9s9" (OuterVolumeSpecName: "kube-api-access-8c9s9") pod "4e565a65-2e13-4acf-8a05-c2a51bd97425" (UID: "4e565a65-2e13-4acf-8a05-c2a51bd97425"). InnerVolumeSpecName "kube-api-access-8c9s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:28.809812 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.809730    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e565a65-2e13-4acf-8a05-c2a51bd97425-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:04:28.809812 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.809760    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e565a65-2e13-4acf-8a05-c2a51bd97425-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:04:28.809812 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:28.809770    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c9s9\" (UniqueName: \"kubernetes.io/projected/4e565a65-2e13-4acf-8a05-c2a51bd97425-kube-api-access-8c9s9\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:04:29.016824 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:29.016795    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"]
Jul 28 18:04:29.026148 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:29.026119    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48"]
Jul 28 18:04:29.181079 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:29.180992    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e565a65-2e13-4acf-8a05-c2a51bd97425" path="/var/lib/kubelet/pods/4e565a65-2e13-4acf-8a05-c2a51bd97425/volumes"
Jul 28 18:04:29.589394 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:04:29.589346    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pfx48" podUID="4e565a65-2e13-4acf-8a05-c2a51bd97425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 18:05:23.700186 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.700156    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"]
Jul 28 18:05:23.700669 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.700454    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e565a65-2e13-4acf-8a05-c2a51bd97425" containerName="main"
Jul 28 18:05:23.700669 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.700466    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e565a65-2e13-4acf-8a05-c2a51bd97425" containerName="main"
Jul 28 18:05:23.700669 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.700515    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e565a65-2e13-4acf-8a05-c2a51bd97425" containerName="main"
Jul 28 18:05:23.703443 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.703426    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.715019 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:23.714992    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-bkrf6\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-5bc7dccd\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-bkrf6\"" type="*v1.Secret"
Jul 28 18:05:23.721104 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:23.721074    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-5bc7dccd\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 28 18:05:23.721104 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:23.721091    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-5bc7dccd\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\"" type="*v1.ConfigMap"
Jul 28 18:05:23.721260 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:23.721075    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-5bc7dccd\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\"" type="*v1.Secret"
Jul 28 18:05:23.722194 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:23.722173    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-5bc7dccd\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\"" type="*v1.Secret"
Jul 28 18:05:23.722262 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:23.722239    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-5bc7dccd\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.722903 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:23.722883    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-20.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-5bc7dccd\": no relationship found between node 'ip-10-0-135-20.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 28 18:05:23.756737 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.756709    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"]
Jul 28 18:05:23.763298 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.763410 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763310    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c84d121-9daa-4d7c-b470-5d3e9431d996-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.763453 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763424    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.763496 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763462    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.763537 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763510    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c84d121-9daa-4d7c-b470-5d3e9431d996-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.763628 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763611    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.763666 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763641    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.763719 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.763686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbf49\" (UniqueName: \"kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864416 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864378    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864416 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864415    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864705 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864438    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c84d121-9daa-4d7c-b470-5d3e9431d996-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864705 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864705 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864488    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864705 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbf49\" (UniqueName: \"kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864705 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864705 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c84d121-9daa-4d7c-b470-5d3e9431d996-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864989 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864926    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.864989 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864950    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.865064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.864994    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.865064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.865024    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:23.866800 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:23.866782    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:24.761030 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:24.760988    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 18:05:24.767332 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:24.767299    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c84d121-9daa-4d7c-b470-5d3e9431d996-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:24.774091 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:24.774068    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:05:24.775560 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:24.775531    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c84d121-9daa-4d7c-b470-5d3e9431d996-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:24.884128 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:24.884098    2585 projected.go:291] Couldn't get configMap e2e-test-llm-inference-service-5bc7dccd/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 28 18:05:24.928674 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:24.928651    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 28 18:05:24.950886 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:24.950865    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 18:05:25.101624 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:25.101591    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-bkrf6\""
Jul 28 18:05:25.243720 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:25.243694    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 18:05:25.244708 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:25.244688    2585 projected.go:196] Error preparing data for projected volume kube-api-access-pbf49 for pod e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc: failed to sync configmap cache: timed out waiting for the condition
Jul 28 18:05:25.244771 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:05:25.244758    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49 podName:0c84d121-9daa-4d7c-b470-5d3e9431d996 nodeName:}" failed. No retries permitted until 2026-07-28 18:05:25.744738417 +0000 UTC m=+2911.088330675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pbf49" (UniqueName: "kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49") pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996") : failed to sync configmap cache: timed out waiting for the condition
Jul 28 18:05:25.780458 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:25.780410    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbf49\" (UniqueName: \"kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:25.783268 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:25.783241    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbf49\" (UniqueName: \"kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:25.813020 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:25.812986    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:26.009620 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:26.009592    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"]
Jul 28 18:05:26.013946 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:05:26.013655    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c84d121_9daa_4d7c_b470_5d3e9431d996.slice/crio-4c6a82c7e6c33c4bbab6f088d490e30034aec4781fe78810354fab8bdbeb0f09 WatchSource:0}: Error finding container 4c6a82c7e6c33c4bbab6f088d490e30034aec4781fe78810354fab8bdbeb0f09: Status 404 returned error can't find the container with id 4c6a82c7e6c33c4bbab6f088d490e30034aec4781fe78810354fab8bdbeb0f09
Jul 28 18:05:26.928212 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:26.928169    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" event={"ID":"0c84d121-9daa-4d7c-b470-5d3e9431d996","Type":"ContainerStarted","Data":"3615351571c7215e7b8910e792626bb67b2ed2978a357da202ca7e6c9e518e99"}
Jul 28 18:05:26.928648 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:26.928237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" event={"ID":"0c84d121-9daa-4d7c-b470-5d3e9431d996","Type":"ContainerStarted","Data":"4c6a82c7e6c33c4bbab6f088d490e30034aec4781fe78810354fab8bdbeb0f09"}
Jul 28 18:05:30.949683 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:30.949642    2585 generic.go:356] "Generic (PLEG): container finished" podID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerID="3615351571c7215e7b8910e792626bb67b2ed2978a357da202ca7e6c9e518e99" exitCode=0
Jul 28 18:05:30.950160 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:30.949687    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" event={"ID":"0c84d121-9daa-4d7c-b470-5d3e9431d996","Type":"ContainerDied","Data":"3615351571c7215e7b8910e792626bb67b2ed2978a357da202ca7e6c9e518e99"}
Jul 28 18:05:31.955968 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:31.955928    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" event={"ID":"0c84d121-9daa-4d7c-b470-5d3e9431d996","Type":"ContainerStarted","Data":"389a780efca63d6565871d793bf27098ac9f9e9e5c7f9702cc3880c856bbe284"}
Jul 28 18:05:32.016154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:32.016090    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podStartSLOduration=4.348563771 podStartE2EDuration="9.01606883s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.667505059s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:32.011926181 +0000 UTC m=+2917.355518460" watchObservedRunningTime="2026-07-28 18:05:32.01606883 +0000 UTC m=+2917.359661109"
Jul 28 18:05:32.813352 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:32.813305    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:35.814114 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:35.814074    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:05:35.815724 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:35.815693    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 18:05:45.813635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:45.813596    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 18:05:55.813402 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:05:55.813359    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 18:06:05.813388 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:05.813341    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 18:06:15.814309 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:15.814268    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 18:06:25.813583 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:25.813348    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 18:06:35.816320 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:35.815781    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 18:06:45.824053 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:45.824024    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:06:45.831814 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:45.831791    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:06:55.242975 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:55.242858    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:06:55.249700 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:55.249679    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:06:55.373841 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:06:55.373804    2585 scope.go:117] "RemoveContainer" containerID="bb3c4fc75e8d20cd5f1ffb5b22a74c388782cf5c1bad497ccf728c18046fea5b"
Jul 28 18:07:11.801081 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:11.801039    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"]
Jul 28 18:07:26.822903 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:26.822821    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main" containerID="cri-o://389a780efca63d6565871d793bf27098ac9f9e9e5c7f9702cc3880c856bbe284" gracePeriod=45
Jul 28 18:07:27.503142 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.503103    2585 generic.go:356] "Generic (PLEG): container finished" podID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerID="389a780efca63d6565871d793bf27098ac9f9e9e5c7f9702cc3880c856bbe284" exitCode=0
Jul 28 18:07:27.503305 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.503175    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" event={"ID":"0c84d121-9daa-4d7c-b470-5d3e9431d996","Type":"ContainerDied","Data":"389a780efca63d6565871d793bf27098ac9f9e9e5c7f9702cc3880c856bbe284"}
Jul 28 18:07:27.503305 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.503241    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc" event={"ID":"0c84d121-9daa-4d7c-b470-5d3e9431d996","Type":"ContainerDied","Data":"4c6a82c7e6c33c4bbab6f088d490e30034aec4781fe78810354fab8bdbeb0f09"}
Jul 28 18:07:27.503305 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.503266    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c6a82c7e6c33c4bbab6f088d490e30034aec4781fe78810354fab8bdbeb0f09"
Jul 28 18:07:27.503487 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.503411    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:07:27.598660 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598628    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-tmp-dir\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.598811 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598682    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c84d121-9daa-4d7c-b470-5d3e9431d996-tls-certs\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.598811 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598708    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-home\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.598811 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598725    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-kserve-provision-location\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.598811 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598750    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-model-cache\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.598811 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598777    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c84d121-9daa-4d7c-b470-5d3e9431d996-cabundle-cert\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.598811 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598799    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbf49\" (UniqueName: \"kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.599134 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.598834    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-dshm\") pod \"0c84d121-9daa-4d7c-b470-5d3e9431d996\" (UID: \"0c84d121-9daa-4d7c-b470-5d3e9431d996\") "
Jul 28 18:07:27.599188 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.599127    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:27.599239 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.599204    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c84d121-9daa-4d7c-b470-5d3e9431d996-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:07:27.599598 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.599571    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-home" (OuterVolumeSpecName: "home") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:27.601221 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.601190    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-dshm" (OuterVolumeSpecName: "dshm") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:27.601318 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.601273    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49" (OuterVolumeSpecName: "kube-api-access-pbf49") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "kube-api-access-pbf49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:07:27.601365 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.601352    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c84d121-9daa-4d7c-b470-5d3e9431d996-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:07:27.610096 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.610069    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:27.668322 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.668283    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c84d121-9daa-4d7c-b470-5d3e9431d996" (UID: "0c84d121-9daa-4d7c-b470-5d3e9431d996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:27.699915 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699883    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.699915 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699911    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c84d121-9daa-4d7c-b470-5d3e9431d996-cabundle-cert\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.700107 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699925    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbf49\" (UniqueName: \"kubernetes.io/projected/0c84d121-9daa-4d7c-b470-5d3e9431d996-kube-api-access-pbf49\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.700107 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699935    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.700107 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699944    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.700107 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699952    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c84d121-9daa-4d7c-b470-5d3e9431d996-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.700107 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699960    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.700107 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:27.699968    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c84d121-9daa-4d7c-b470-5d3e9431d996-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:07:28.507445 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:28.507413    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"
Jul 28 18:07:28.535765 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:28.531806    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"]
Jul 28 18:07:28.537902 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:28.537871    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9fccb4ff-nhsrc"]
Jul 28 18:07:29.179938 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:29.179906    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" path="/var/lib/kubelet/pods/0c84d121-9daa-4d7c-b470-5d3e9431d996/volumes"
Jul 28 18:07:44.924124 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.924088    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"]
Jul 28 18:07:44.924627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.924373    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="storage-initializer"
Jul 28 18:07:44.924627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.924384    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="storage-initializer"
Jul 28 18:07:44.924627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.924406    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main"
Jul 28 18:07:44.924627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.924411    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main"
Jul 28 18:07:44.924627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.924479    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c84d121-9daa-4d7c-b470-5d3e9431d996" containerName="main"
Jul 28 18:07:44.927538 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.927517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:44.930528 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.930489    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-jljxf\""
Jul 28 18:07:44.930661 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.930621    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 18:07:44.930661 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.930621    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 18:07:44.930932 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.930914    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:07:44.931885 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.931862    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 28 18:07:44.931885 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.931853    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 18:07:44.945393 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:44.945371    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"]
Jul 28 18:07:45.031669 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031620    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p996s\" (UniqueName: \"kubernetes.io/projected/7933a5f5-5d5d-4b32-b87f-c8906892af29-kube-api-access-p996s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.031849 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.031849 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031702    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.031849 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031723    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7933a5f5-5d5d-4b32-b87f-c8906892af29-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.031849 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031752    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7933a5f5-5d5d-4b32-b87f-c8906892af29-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.031849 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031791    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.031849 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031815    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.031849 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.031832    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133224 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133192    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7933a5f5-5d5d-4b32-b87f-c8906892af29-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133418 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133238    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133418 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133267    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133418 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133285    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133418 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133316    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p996s\" (UniqueName: \"kubernetes.io/projected/7933a5f5-5d5d-4b32-b87f-c8906892af29-kube-api-access-p996s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133418 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133350    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133418 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133418 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133397    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7933a5f5-5d5d-4b32-b87f-c8906892af29-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133837 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133806    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133895 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133876    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.133999 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.133976    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.134137 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.134000    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.134137 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.134132    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7933a5f5-5d5d-4b32-b87f-c8906892af29-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.135656 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.135633    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.135835 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.135819    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7933a5f5-5d5d-4b32-b87f-c8906892af29-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.143360 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.143329    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p996s\" (UniqueName: \"kubernetes.io/projected/7933a5f5-5d5d-4b32-b87f-c8906892af29-kube-api-access-p996s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.237825 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.237744    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:07:45.376192 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.375956    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"]
Jul 28 18:07:45.378425 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:07:45.378398    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7933a5f5_5d5d_4b32_b87f_c8906892af29.slice/crio-eeba808d5de255ba148b84027e7d30b6ebbd7e9d424f43c0c3bae8db226ebc94 WatchSource:0}: Error finding container eeba808d5de255ba148b84027e7d30b6ebbd7e9d424f43c0c3bae8db226ebc94: Status 404 returned error can't find the container with id eeba808d5de255ba148b84027e7d30b6ebbd7e9d424f43c0c3bae8db226ebc94
Jul 28 18:07:45.579729 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.579688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" event={"ID":"7933a5f5-5d5d-4b32-b87f-c8906892af29","Type":"ContainerStarted","Data":"cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a"}
Jul 28 18:07:45.579917 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:45.579752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" event={"ID":"7933a5f5-5d5d-4b32-b87f-c8906892af29","Type":"ContainerStarted","Data":"eeba808d5de255ba148b84027e7d30b6ebbd7e9d424f43c0c3bae8db226ebc94"}
Jul 28 18:07:49.417978 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:49.417898    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"]
Jul 28 18:07:52.739171 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:07:52.739133    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" podUID="63d4cb62-d860-4817-81f3-a3bdb26f689f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:08:00.644513 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:00.644421    2585 generic.go:356] "Generic (PLEG): container finished" podID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerID="cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a" exitCode=0
Jul 28 18:08:00.644513 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:00.644475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" event={"ID":"7933a5f5-5d5d-4b32-b87f-c8906892af29","Type":"ContainerDied","Data":"cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a"}
Jul 28 18:08:01.651642 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:01.651607    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" event={"ID":"7933a5f5-5d5d-4b32-b87f-c8906892af29","Type":"ContainerStarted","Data":"4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b"}
Jul 28 18:08:01.679892 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:01.679840    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podStartSLOduration=2.845522941 podStartE2EDuration="17.679824461s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.83430152s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:01.677913446 +0000 UTC m=+3067.021505734" watchObservedRunningTime="2026-07-28 18:08:01.679824461 +0000 UTC m=+3067.023416737"
Jul 28 18:08:02.238196 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:02.238164    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:08:02.739163 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:02.739114    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" podUID="63d4cb62-d860-4817-81f3-a3bdb26f689f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:08:04.439282 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.439238    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" podUID="63d4cb62-d860-4817-81f3-a3bdb26f689f" containerName="main" containerID="cri-o://2a10da0b57c1579b0dab49e0e1bfbd6140a39afc653fd97341e87ff0e292dff8" gracePeriod=45
Jul 28 18:08:04.667223 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.667189    2585 generic.go:356] "Generic (PLEG): container finished" podID="63d4cb62-d860-4817-81f3-a3bdb26f689f" containerID="2a10da0b57c1579b0dab49e0e1bfbd6140a39afc653fd97341e87ff0e292dff8" exitCode=0
Jul 28 18:08:04.667386 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.667257    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" event={"ID":"63d4cb62-d860-4817-81f3-a3bdb26f689f","Type":"ContainerDied","Data":"2a10da0b57c1579b0dab49e0e1bfbd6140a39afc653fd97341e87ff0e292dff8"}
Jul 28 18:08:04.707793 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.707732    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 18:08:04.815820 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.815783    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d4cb62-d860-4817-81f3-a3bdb26f689f-tls-certs\") pod \"63d4cb62-d860-4817-81f3-a3bdb26f689f\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") "
Jul 28 18:08:04.816008 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.815857    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qfcd\" (UniqueName: \"kubernetes.io/projected/63d4cb62-d860-4817-81f3-a3bdb26f689f-kube-api-access-4qfcd\") pod \"63d4cb62-d860-4817-81f3-a3bdb26f689f\" (UID: \"63d4cb62-d860-4817-81f3-a3bdb26f689f\") "
Jul 28 18:08:04.818044 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.818012    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d4cb62-d860-4817-81f3-a3bdb26f689f-kube-api-access-4qfcd" (OuterVolumeSpecName: "kube-api-access-4qfcd") pod "63d4cb62-d860-4817-81f3-a3bdb26f689f" (UID: "63d4cb62-d860-4817-81f3-a3bdb26f689f"). InnerVolumeSpecName "kube-api-access-4qfcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:08:04.818044 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.818032    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d4cb62-d860-4817-81f3-a3bdb26f689f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63d4cb62-d860-4817-81f3-a3bdb26f689f" (UID: "63d4cb62-d860-4817-81f3-a3bdb26f689f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:08:04.917345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.917298    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d4cb62-d860-4817-81f3-a3bdb26f689f-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:08:04.917345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:04.917343    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qfcd\" (UniqueName: \"kubernetes.io/projected/63d4cb62-d860-4817-81f3-a3bdb26f689f-kube-api-access-4qfcd\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:08:05.237846 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:05.237815    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:08:05.239447 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:05.239416    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:08:05.673268 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:05.673236    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"
Jul 28 18:08:05.673268 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:05.673254    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm" event={"ID":"63d4cb62-d860-4817-81f3-a3bdb26f689f","Type":"ContainerDied","Data":"a624a55c154befdcb914627b335761eaafb604b1a54951112851c721fc9a1688"}
Jul 28 18:08:05.673888 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:05.673313    2585 scope.go:117] "RemoveContainer" containerID="2a10da0b57c1579b0dab49e0e1bfbd6140a39afc653fd97341e87ff0e292dff8"
Jul 28 18:08:05.690242 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:05.690218    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"]
Jul 28 18:08:05.695429 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:05.695406    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm"]
Jul 28 18:08:07.181136 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:07.181101    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d4cb62-d860-4817-81f3-a3bdb26f689f" path="/var/lib/kubelet/pods/63d4cb62-d860-4817-81f3-a3bdb26f689f/volumes"
Jul 28 18:08:15.238340 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:15.238302    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:08:25.238295 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:25.238243    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:08:35.238491 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:35.238445    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:08:45.238874 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:45.238821    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:08:55.238539 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:08:55.238494    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:09:05.238693 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:05.238651    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:09:15.248055 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:15.248015    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:09:15.255983 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:15.255956    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:09:16.865137 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:16.865058    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"]
Jul 28 18:09:32.003686 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.003636    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" containerID="cri-o://4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b" gracePeriod=45
Jul 28 18:09:32.249204 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.249156    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 18:09:32.648017 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.647988    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:09:32.781911 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.781868    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-home\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782114 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.781937    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7933a5f5-5d5d-4b32-b87f-c8906892af29-cabundle-cert\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782114 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.781970    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7933a5f5-5d5d-4b32-b87f-c8906892af29-tls-certs\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782114 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782005    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-tmp-dir\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782114 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782040    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-model-cache\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782114 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782089    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p996s\" (UniqueName: \"kubernetes.io/projected/7933a5f5-5d5d-4b32-b87f-c8906892af29-kube-api-access-p996s\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782114 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782109    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-dshm\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782420 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782141    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-kserve-provision-location\") pod \"7933a5f5-5d5d-4b32-b87f-c8906892af29\" (UID: \"7933a5f5-5d5d-4b32-b87f-c8906892af29\") "
Jul 28 18:09:32.782420 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782367    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:09:32.782420 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782392    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-home" (OuterVolumeSpecName: "home") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:09:32.782420 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782373    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-model-cache" (OuterVolumeSpecName: "model-cache") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:09:32.782667 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.782618    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7933a5f5-5d5d-4b32-b87f-c8906892af29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:09:32.785022 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.784348    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7933a5f5-5d5d-4b32-b87f-c8906892af29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:09:32.785022 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.784689    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-dshm" (OuterVolumeSpecName: "dshm") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:09:32.785022 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.784775    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7933a5f5-5d5d-4b32-b87f-c8906892af29-kube-api-access-p996s" (OuterVolumeSpecName: "kube-api-access-p996s") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "kube-api-access-p996s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:09:32.851864 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.851835    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7933a5f5-5d5d-4b32-b87f-c8906892af29" (UID: "7933a5f5-5d5d-4b32-b87f-c8906892af29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:09:32.883076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883015    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.883076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883039    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.883076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883050    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p996s\" (UniqueName: \"kubernetes.io/projected/7933a5f5-5d5d-4b32-b87f-c8906892af29-kube-api-access-p996s\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.883076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883060    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.883076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883069    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.883076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883077    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7933a5f5-5d5d-4b32-b87f-c8906892af29-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.883316 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883085    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7933a5f5-5d5d-4b32-b87f-c8906892af29-cabundle-cert\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.883316 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:32.883093    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7933a5f5-5d5d-4b32-b87f-c8906892af29-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:33.048198 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.048168    2585 generic.go:356] "Generic (PLEG): container finished" podID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerID="4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b" exitCode=0
Jul 28 18:09:33.048606 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.048222    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" event={"ID":"7933a5f5-5d5d-4b32-b87f-c8906892af29","Type":"ContainerDied","Data":"4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b"}
Jul 28 18:09:33.048606 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.048281    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr" event={"ID":"7933a5f5-5d5d-4b32-b87f-c8906892af29","Type":"ContainerDied","Data":"eeba808d5de255ba148b84027e7d30b6ebbd7e9d424f43c0c3bae8db226ebc94"}
Jul 28 18:09:33.048606 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.048225    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"
Jul 28 18:09:33.048606 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.048303    2585 scope.go:117] "RemoveContainer" containerID="4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b"
Jul 28 18:09:33.057672 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.057651    2585 scope.go:117] "RemoveContainer" containerID="cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a"
Jul 28 18:09:33.065352 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.065335    2585 scope.go:117] "RemoveContainer" containerID="4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b"
Jul 28 18:09:33.065599 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:09:33.065581    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b\": container with ID starting with 4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b not found: ID does not exist" containerID="4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b"
Jul 28 18:09:33.065678 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.065604    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b"} err="failed to get container status \"4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b\": rpc error: code = NotFound desc = could not find container \"4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b\": container with ID starting with 4b798521b811296ce39d4bfc38154c73c417872da14d1a4aeeb82536fe7e520b not found: ID does not exist"
Jul 28 18:09:33.065678 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.065621    2585 scope.go:117] "RemoveContainer" containerID="cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a"
Jul 28 18:09:33.065841 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:09:33.065822    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a\": container with ID starting with cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a not found: ID does not exist" containerID="cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a"
Jul 28 18:09:33.065894 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.065849    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a"} err="failed to get container status \"cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a\": rpc error: code = NotFound desc = could not find container \"cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a\": container with ID starting with cd27378450090531bad4b16ad57d95cc0e1d00f373ee61f8cd0ad3fc0251142a not found: ID does not exist"
Jul 28 18:09:33.072372 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.072349    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"]
Jul 28 18:09:33.077376 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.077355    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6bdcd5f5cd-xb4vr"]
Jul 28 18:09:33.180098 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:33.180033    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" path="/var/lib/kubelet/pods/7933a5f5-5d5d-4b32-b87f-c8906892af29/volumes"
Jul 28 18:09:40.088953 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.088914    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"]
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089191    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="storage-initializer"
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089202    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="storage-initializer"
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089218    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d4cb62-d860-4817-81f3-a3bdb26f689f" containerName="main"
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089224    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d4cb62-d860-4817-81f3-a3bdb26f689f" containerName="main"
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089235    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main"
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089241    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main"
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089292    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d4cb62-d860-4817-81f3-a3bdb26f689f" containerName="main"
Jul 28 18:09:40.089351 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.089299    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7933a5f5-5d5d-4b32-b87f-c8906892af29" containerName="main"
Jul 28 18:09:40.091235 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.091210    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:40.094891 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.094866    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 18:09:40.096022 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.096000    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-4grpt\""
Jul 28 18:09:40.096126 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.096000    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 18:09:40.099531 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.099513    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"]
Jul 28 18:09:40.244131 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.244100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdh9\" (UniqueName: \"kubernetes.io/projected/5790585c-ec62-43c7-8fc6-fe883aad84e6-kube-api-access-gtdh9\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:40.344947 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.344847    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdh9\" (UniqueName: \"kubernetes.io/projected/5790585c-ec62-43c7-8fc6-fe883aad84e6-kube-api-access-gtdh9\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:40.353986 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:40.353963    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdh9\" (UniqueName: \"kubernetes.io/projected/5790585c-ec62-43c7-8fc6-fe883aad84e6-kube-api-access-gtdh9\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:42.562592 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:42.562533    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:42.568518 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:42.568495    2585 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:42.663736 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:42.663702    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:42.663918 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:42.663846    2585 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:42.667183 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:42.667162    2585 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") DevicePath \"csi-78d3d6469687274dd7bc237f792822bf0febc8cb28bfc32ee0d58aaeb2ddb269\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:42.822241 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:42.822167    2585 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6a5e352080a1d2f0ac8b9a6251da2eb9bfe49c663bcfb9bbae16f74646e02340/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:42.836409 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:42.836382    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"e2e-pvc-model-download-6f2vr\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:43.101797 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:43.101714    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:43.224993 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:43.224969    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"]
Jul 28 18:09:43.226778 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:09:43.226748    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5790585c_ec62_43c7_8fc6_fe883aad84e6.slice/crio-357a14ccca8be16fa0009db02592f73ba9fbf0eefaeecf071b49db8a2b60b68a WatchSource:0}: Error finding container 357a14ccca8be16fa0009db02592f73ba9fbf0eefaeecf071b49db8a2b60b68a: Status 404 returned error can't find the container with id 357a14ccca8be16fa0009db02592f73ba9fbf0eefaeecf071b49db8a2b60b68a
Jul 28 18:09:44.097144 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:44.096968    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr" event={"ID":"5790585c-ec62-43c7-8fc6-fe883aad84e6","Type":"ContainerStarted","Data":"357745ffe1cc3c8bf7ebb3067e89471d03931d4141b5840afcb35679a2d9b1ae"}
Jul 28 18:09:44.097144 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:44.097037    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr" event={"ID":"5790585c-ec62-43c7-8fc6-fe883aad84e6","Type":"ContainerStarted","Data":"357a14ccca8be16fa0009db02592f73ba9fbf0eefaeecf071b49db8a2b60b68a"}
Jul 28 18:09:44.118520 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:44.118449    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr" podStartSLOduration=7.118421902 podStartE2EDuration="7.118421902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:44.116860128 +0000 UTC m=+3169.460452406" watchObservedRunningTime="2026-07-28 18:09:44.118421902 +0000 UTC m=+3169.462014180"
Jul 28 18:09:50.121301 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:50.121265    2585 generic.go:356] "Generic (PLEG): container finished" podID="5790585c-ec62-43c7-8fc6-fe883aad84e6" containerID="357745ffe1cc3c8bf7ebb3067e89471d03931d4141b5840afcb35679a2d9b1ae" exitCode=0
Jul 28 18:09:50.121724 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:50.121334    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr" event={"ID":"5790585c-ec62-43c7-8fc6-fe883aad84e6","Type":"ContainerDied","Data":"357745ffe1cc3c8bf7ebb3067e89471d03931d4141b5840afcb35679a2d9b1ae"}
Jul 28 18:09:51.267915 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.267886    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:51.342140 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.342110    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"5790585c-ec62-43c7-8fc6-fe883aad84e6\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") "
Jul 28 18:09:51.342313 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.342156    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtdh9\" (UniqueName: \"kubernetes.io/projected/5790585c-ec62-43c7-8fc6-fe883aad84e6-kube-api-access-gtdh9\") pod \"5790585c-ec62-43c7-8fc6-fe883aad84e6\" (UID: \"5790585c-ec62-43c7-8fc6-fe883aad84e6\") "
Jul 28 18:09:51.344333 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.344303    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5790585c-ec62-43c7-8fc6-fe883aad84e6-kube-api-access-gtdh9" (OuterVolumeSpecName: "kube-api-access-gtdh9") pod "5790585c-ec62-43c7-8fc6-fe883aad84e6" (UID: "5790585c-ec62-43c7-8fc6-fe883aad84e6"). InnerVolumeSpecName "kube-api-access-gtdh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:09:51.346356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.346327    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6" (OuterVolumeSpecName: "model-storage") pod "5790585c-ec62-43c7-8fc6-fe883aad84e6" (UID: "5790585c-ec62-43c7-8fc6-fe883aad84e6"). InnerVolumeSpecName "pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:09:51.443475 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.443385    2585 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") on node \"ip-10-0-135-20.ec2.internal\" "
Jul 28 18:09:51.443475 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.443419    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtdh9\" (UniqueName: \"kubernetes.io/projected/5790585c-ec62-43c7-8fc6-fe883aad84e6-kube-api-access-gtdh9\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:09:51.559620 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.559600    2585 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6") on node "ip-10-0-135-20.ec2.internal"
Jul 28 18:09:51.645202 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:51.645167    2585 reconciler_common.go:299] "Volume detached for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"csi-78d3d6469687274dd7bc237f792822bf0febc8cb28bfc32ee0d58aaeb2ddb269\""
Jul 28 18:09:52.130735 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:52.130703    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"
Jul 28 18:09:52.130910 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:52.130707    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr" event={"ID":"5790585c-ec62-43c7-8fc6-fe883aad84e6","Type":"ContainerDied","Data":"357a14ccca8be16fa0009db02592f73ba9fbf0eefaeecf071b49db8a2b60b68a"}
Jul 28 18:09:52.130910 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:09:52.130824    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="357a14ccca8be16fa0009db02592f73ba9fbf0eefaeecf071b49db8a2b60b68a"
Jul 28 18:10:03.398487 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.398452    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"]
Jul 28 18:10:03.398895 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.398766    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5790585c-ec62-43c7-8fc6-fe883aad84e6" containerName="storage-initializer"
Jul 28 18:10:03.398895 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.398778    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5790585c-ec62-43c7-8fc6-fe883aad84e6" containerName="storage-initializer"
Jul 28 18:10:03.405173 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.405122    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5790585c-ec62-43c7-8fc6-fe883aad84e6" containerName="storage-initializer"
Jul 28 18:10:03.410367 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.410341    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.413368 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.413342    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 18:10:03.413368 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.413356    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 18:10:03.413537 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.413344    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 18:10:03.413537 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.413342    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-4grpt\""
Jul 28 18:10:03.418742 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.418721    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"]
Jul 28 18:10:03.544823 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.544782    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.545008 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.544835    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.545008 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.544901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.545008 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.544947    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245f7106-c195-489f-bef8-ee21611da400-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.545008 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.544982    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxfw\" (UniqueName: \"kubernetes.io/projected/245f7106-c195-489f-bef8-ee21611da400-kube-api-access-tdxfw\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.545155 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.545010    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646185 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646148    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646377 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646212    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646377 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646238    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646377 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646377 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646278    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245f7106-c195-489f-bef8-ee21611da400-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646377 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxfw\" (UniqueName: \"kubernetes.io/projected/245f7106-c195-489f-bef8-ee21611da400-kube-api-access-tdxfw\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646674 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646616    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646674 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646632    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.646788 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.646741    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.648620 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.648547    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.648920 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.648896    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245f7106-c195-489f-bef8-ee21611da400-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.655711 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.655691    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxfw\" (UniqueName: \"kubernetes.io/projected/245f7106-c195-489f-bef8-ee21611da400-kube-api-access-tdxfw\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:03.710973 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.710939    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"]
Jul 28 18:10:03.714312 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.714295    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:03.717097 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.717067    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-j6ffm\""
Jul 28 18:10:03.722939 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.722916    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"]
Jul 28 18:10:03.848685 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.848654    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g4fm\" (UniqueName: \"kubernetes.io/projected/18af8672-0bf2-4536-9b73-c94f275a1002-kube-api-access-8g4fm\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:03.848685 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.848691    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18af8672-0bf2-4536-9b73-c94f275a1002-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:03.949295 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.949195    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g4fm\" (UniqueName: \"kubernetes.io/projected/18af8672-0bf2-4536-9b73-c94f275a1002-kube-api-access-8g4fm\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:03.949295 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.949236    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18af8672-0bf2-4536-9b73-c94f275a1002-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:03.951711 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.951686    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18af8672-0bf2-4536-9b73-c94f275a1002-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:03.957646 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:03.957614    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g4fm\" (UniqueName: \"kubernetes.io/projected/18af8672-0bf2-4536-9b73-c94f275a1002-kube-api-access-8g4fm\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:04.024749 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:04.024720    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:04.157017 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:04.156817    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"]
Jul 28 18:10:04.159262 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:10:04.159223    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18af8672_0bf2_4536_9b73_c94f275a1002.slice/crio-5a23cf14841558821b6ec0fc24304351e859cfb9d997ac32e3e600e8955d4973 WatchSource:0}: Error finding container 5a23cf14841558821b6ec0fc24304351e859cfb9d997ac32e3e600e8955d4973: Status 404 returned error can't find the container with id 5a23cf14841558821b6ec0fc24304351e859cfb9d997ac32e3e600e8955d4973
Jul 28 18:10:04.177263 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:04.177218    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts" event={"ID":"18af8672-0bf2-4536-9b73-c94f275a1002","Type":"ContainerStarted","Data":"5a23cf14841558821b6ec0fc24304351e859cfb9d997ac32e3e600e8955d4973"}
Jul 28 18:10:05.183733 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:05.183700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts" event={"ID":"18af8672-0bf2-4536-9b73-c94f275a1002","Type":"ContainerStarted","Data":"892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1"}
Jul 28 18:10:05.184183 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:05.183841    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:11.914841 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:11.914801    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:11.922299 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:11.922269    2585 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:12.015968 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.015929    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:12.016124 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.016061    2585 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:12.019378 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.019360    2585 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") DevicePath \"csi-78d3d6469687274dd7bc237f792822bf0febc8cb28bfc32ee0d58aaeb2ddb269\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:12.110715 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.110683    2585 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6a5e352080a1d2f0ac8b9a6251da2eb9bfe49c663bcfb9bbae16f74646e02340/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:12.127354 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.127329    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:12.422491 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.422454    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:12.572402 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.572300    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts" podStartSLOduration=9.572280951 podStartE2EDuration="9.572280951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:05.20475787 +0000 UTC m=+3190.548350148" watchObservedRunningTime="2026-07-28 18:10:12.572280951 +0000 UTC m=+3197.915873231"
Jul 28 18:10:12.572615 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:12.572597    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"]
Jul 28 18:10:12.575002 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:10:12.574974    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245f7106_c195_489f_bef8_ee21611da400.slice/crio-c0e3eae234760d63feec1207bb3225adf8d14f5ff8917db37a7da707ade08301 WatchSource:0}: Error finding container c0e3eae234760d63feec1207bb3225adf8d14f5ff8917db37a7da707ade08301: Status 404 returned error can't find the container with id c0e3eae234760d63feec1207bb3225adf8d14f5ff8917db37a7da707ade08301
Jul 28 18:10:13.219151 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:13.219105    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" event={"ID":"245f7106-c195-489f-bef8-ee21611da400","Type":"ContainerStarted","Data":"d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8"}
Jul 28 18:10:13.219578 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:13.219159    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" event={"ID":"245f7106-c195-489f-bef8-ee21611da400","Type":"ContainerStarted","Data":"c0e3eae234760d63feec1207bb3225adf8d14f5ff8917db37a7da707ade08301"}
Jul 28 18:10:13.243620 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:13.243506    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podStartSLOduration=10.243487077 podStartE2EDuration="10.243487077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:13.240687564 +0000 UTC m=+3198.584279843" watchObservedRunningTime="2026-07-28 18:10:13.243487077 +0000 UTC m=+3198.587079357"
Jul 28 18:10:13.422697 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:13.422636    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:22.422997 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:22.422945    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:10:22.424945 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:22.424897    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:10:32.423250 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:32.423193    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:10:36.189541 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:36.189505    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:10:42.423376 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:42.423332    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:10:52.423345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:10:52.423236    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:11:02.423671 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:02.423622    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:11:12.422917 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:12.422872    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:11:22.423707 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:22.423660    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:11:32.423668 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:32.423620    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:11:42.423484 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:42.423441    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:11:52.423332 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:52.423283    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:11:55.269132 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:55.269104    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:11:55.278233 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:55.278211    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:11:55.436450 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:55.436416    2585 scope.go:117] "RemoveContainer" containerID="389a780efca63d6565871d793bf27098ac9f9e9e5c7f9702cc3880c856bbe284"
Jul 28 18:11:55.444991 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:11:55.444968    2585 scope.go:117] "RemoveContainer" containerID="3615351571c7215e7b8910e792626bb67b2ed2978a357da202ca7e6c9e518e99"
Jul 28 18:12:02.433544 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:02.433511    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:12:02.441545 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:02.441526    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:12:07.316623 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:07.310994    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"]
Jul 28 18:12:07.316623 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:07.312786    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"]
Jul 28 18:12:16.189165 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:16.189130    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts" podUID="18af8672-0bf2-4536-9b73-c94f275a1002" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:12:22.339118 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.338999    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts" podUID="18af8672-0bf2-4536-9b73-c94f275a1002" containerName="main" containerID="cri-o://892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1" gracePeriod=45
Jul 28 18:12:22.342137 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.342048    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" containerID="cri-o://d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8" gracePeriod=45
Jul 28 18:12:22.434584 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.434515    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 28 18:12:22.588925 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.588899    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:12:22.718990 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.718885    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18af8672-0bf2-4536-9b73-c94f275a1002-tls-certs\") pod \"18af8672-0bf2-4536-9b73-c94f275a1002\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") "
Jul 28 18:12:22.718990 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.718957    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g4fm\" (UniqueName: \"kubernetes.io/projected/18af8672-0bf2-4536-9b73-c94f275a1002-kube-api-access-8g4fm\") pod \"18af8672-0bf2-4536-9b73-c94f275a1002\" (UID: \"18af8672-0bf2-4536-9b73-c94f275a1002\") "
Jul 28 18:12:22.721046 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.721004    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18af8672-0bf2-4536-9b73-c94f275a1002-kube-api-access-8g4fm" (OuterVolumeSpecName: "kube-api-access-8g4fm") pod "18af8672-0bf2-4536-9b73-c94f275a1002" (UID: "18af8672-0bf2-4536-9b73-c94f275a1002"). InnerVolumeSpecName "kube-api-access-8g4fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:12:22.721163 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.721107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18af8672-0bf2-4536-9b73-c94f275a1002-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18af8672-0bf2-4536-9b73-c94f275a1002" (UID: "18af8672-0bf2-4536-9b73-c94f275a1002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:12:22.819327 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.819295    2585 generic.go:356] "Generic (PLEG): container finished" podID="18af8672-0bf2-4536-9b73-c94f275a1002" containerID="892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1" exitCode=0
Jul 28 18:12:22.819527 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.819357    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"
Jul 28 18:12:22.819527 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.819376    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts" event={"ID":"18af8672-0bf2-4536-9b73-c94f275a1002","Type":"ContainerDied","Data":"892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1"}
Jul 28 18:12:22.819527 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.819459    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts" event={"ID":"18af8672-0bf2-4536-9b73-c94f275a1002","Type":"ContainerDied","Data":"5a23cf14841558821b6ec0fc24304351e859cfb9d997ac32e3e600e8955d4973"}
Jul 28 18:12:22.819527 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.819483    2585 scope.go:117] "RemoveContainer" containerID="892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1"
Jul 28 18:12:22.819795 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.819624    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g4fm\" (UniqueName: \"kubernetes.io/projected/18af8672-0bf2-4536-9b73-c94f275a1002-kube-api-access-8g4fm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:22.819795 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.819651    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18af8672-0bf2-4536-9b73-c94f275a1002-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:22.828507 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.828488    2585 scope.go:117] "RemoveContainer" containerID="892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1"
Jul 28 18:12:22.828832 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:12:22.828811    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1\": container with ID starting with 892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1 not found: ID does not exist" containerID="892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1"
Jul 28 18:12:22.828900 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.828845    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1"} err="failed to get container status \"892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1\": rpc error: code = NotFound desc = could not find container \"892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1\": container with ID starting with 892bc99006a76b6586d5423265af952f130d24b70ba9701e2fde0cdbc36386f1 not found: ID does not exist"
Jul 28 18:12:22.845706 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.845675    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"]
Jul 28 18:12:22.850879 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:22.850854    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schennnts"]
Jul 28 18:12:23.087600 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.087546    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:12:23.180223 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.180189    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18af8672-0bf2-4536-9b73-c94f275a1002" path="/var/lib/kubelet/pods/18af8672-0bf2-4536-9b73-c94f275a1002/volumes"
Jul 28 18:12:23.223830 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.223799    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdxfw\" (UniqueName: \"kubernetes.io/projected/245f7106-c195-489f-bef8-ee21611da400-kube-api-access-tdxfw\") pod \"245f7106-c195-489f-bef8-ee21611da400\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") "
Jul 28 18:12:23.224005 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.223844    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-tmp-dir\") pod \"245f7106-c195-489f-bef8-ee21611da400\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") "
Jul 28 18:12:23.224005 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.223868    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245f7106-c195-489f-bef8-ee21611da400-tls-certs\") pod \"245f7106-c195-489f-bef8-ee21611da400\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") "
Jul 28 18:12:23.224005 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.223892    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-dshm\") pod \"245f7106-c195-489f-bef8-ee21611da400\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") "
Jul 28 18:12:23.224186 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.224000    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-model-cache\") pod \"245f7106-c195-489f-bef8-ee21611da400\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") "
Jul 28 18:12:23.224186 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.224046    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-home\") pod \"245f7106-c195-489f-bef8-ee21611da400\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") "
Jul 28 18:12:23.224186 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.224160    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") pod \"245f7106-c195-489f-bef8-ee21611da400\" (UID: \"245f7106-c195-489f-bef8-ee21611da400\") "
Jul 28 18:12:23.224350 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.224285    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-model-cache" (OuterVolumeSpecName: "model-cache") pod "245f7106-c195-489f-bef8-ee21611da400" (UID: "245f7106-c195-489f-bef8-ee21611da400"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:12:23.224473 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.224446    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:23.224838 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.224809    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-home" (OuterVolumeSpecName: "home") pod "245f7106-c195-489f-bef8-ee21611da400" (UID: "245f7106-c195-489f-bef8-ee21611da400"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:12:23.226176 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.226145    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245f7106-c195-489f-bef8-ee21611da400-kube-api-access-tdxfw" (OuterVolumeSpecName: "kube-api-access-tdxfw") pod "245f7106-c195-489f-bef8-ee21611da400" (UID: "245f7106-c195-489f-bef8-ee21611da400"). InnerVolumeSpecName "kube-api-access-tdxfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:12:23.226262 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.226205    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-dshm" (OuterVolumeSpecName: "dshm") pod "245f7106-c195-489f-bef8-ee21611da400" (UID: "245f7106-c195-489f-bef8-ee21611da400"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:12:23.226382 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.226368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245f7106-c195-489f-bef8-ee21611da400-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "245f7106-c195-489f-bef8-ee21611da400" (UID: "245f7106-c195-489f-bef8-ee21611da400"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:12:23.227196 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.227177    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6" (OuterVolumeSpecName: "kserve-pvc-source") pod "245f7106-c195-489f-bef8-ee21611da400" (UID: "245f7106-c195-489f-bef8-ee21611da400"). InnerVolumeSpecName "pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:12:23.235831 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.235808    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "245f7106-c195-489f-bef8-ee21611da400" (UID: "245f7106-c195-489f-bef8-ee21611da400"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:12:23.325816 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.325784    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxfw\" (UniqueName: \"kubernetes.io/projected/245f7106-c195-489f-bef8-ee21611da400-kube-api-access-tdxfw\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:23.325816 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.325814    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:23.326014 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.325824    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245f7106-c195-489f-bef8-ee21611da400-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:23.326014 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.325834    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:23.326014 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.325841    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245f7106-c195-489f-bef8-ee21611da400-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:23.326014 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.325867    2585 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") on node \"ip-10-0-135-20.ec2.internal\" "
Jul 28 18:12:23.352895 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.352813    2585 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6") on node "ip-10-0-135-20.ec2.internal"
Jul 28 18:12:23.426220 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.426189    2585 reconciler_common.go:299] "Volume detached for volume \"pvc-8509ff2c-e0e7-40b4-b960-fb42e73b5e86\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-046ae3e7b03137ec6\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"csi-78d3d6469687274dd7bc237f792822bf0febc8cb28bfc32ee0d58aaeb2ddb269\""
Jul 28 18:12:23.825845 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.825816    2585 generic.go:356] "Generic (PLEG): container finished" podID="245f7106-c195-489f-bef8-ee21611da400" containerID="d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8" exitCode=0
Jul 28 18:12:23.826045 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.825858    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"
Jul 28 18:12:23.826045 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.825898    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" event={"ID":"245f7106-c195-489f-bef8-ee21611da400","Type":"ContainerDied","Data":"d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8"}
Jul 28 18:12:23.826045 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.825955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn" event={"ID":"245f7106-c195-489f-bef8-ee21611da400","Type":"ContainerDied","Data":"c0e3eae234760d63feec1207bb3225adf8d14f5ff8917db37a7da707ade08301"}
Jul 28 18:12:23.826045 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.825985    2585 scope.go:117] "RemoveContainer" containerID="d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8"
Jul 28 18:12:23.835122 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.835099    2585 scope.go:117] "RemoveContainer" containerID="d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8"
Jul 28 18:12:23.835381 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:12:23.835360    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8\": container with ID starting with d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8 not found: ID does not exist" containerID="d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8"
Jul 28 18:12:23.835449 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.835392    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8"} err="failed to get container status \"d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8\": rpc error: code = NotFound desc = could not find container \"d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8\": container with ID starting with d0dd417f2246e97714f007046423ce6d8a330733cf47594ee66eade233841ec8 not found: ID does not exist"
Jul 28 18:12:23.857099 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.857074    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"]
Jul 28 18:12:23.861647 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:23.861626    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnmfn"]
Jul 28 18:12:25.180379 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:25.180347    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="245f7106-c195-489f-bef8-ee21611da400" path="/var/lib/kubelet/pods/245f7106-c195-489f-bef8-ee21611da400/volumes"
Jul 28 18:12:30.290950 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.290916    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"]
Jul 28 18:12:30.291356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.291209    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18af8672-0bf2-4536-9b73-c94f275a1002" containerName="main"
Jul 28 18:12:30.291356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.291220    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="18af8672-0bf2-4536-9b73-c94f275a1002" containerName="main"
Jul 28 18:12:30.291356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.291242    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main"
Jul 28 18:12:30.291356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.291249    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main"
Jul 28 18:12:30.291356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.291302    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="18af8672-0bf2-4536-9b73-c94f275a1002" containerName="main"
Jul 28 18:12:30.291356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.291310    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="245f7106-c195-489f-bef8-ee21611da400" containerName="main"
Jul 28 18:12:30.295646 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.295630    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:30.298819 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.298798    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 18:12:30.300054 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.300034    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-w6gcf\""
Jul 28 18:12:30.300145 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.300071    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 18:12:30.303393 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.303369    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"]
Jul 28 18:12:30.384175 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.384141    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz229\" (UniqueName: \"kubernetes.io/projected/aab63c87-1d86-4b61-ab5d-2bc8ad087853-kube-api-access-bz229\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:30.485362 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.485333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz229\" (UniqueName: \"kubernetes.io/projected/aab63c87-1d86-4b61-ab5d-2bc8ad087853-kube-api-access-bz229\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:30.495183 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:30.495155    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz229\" (UniqueName: \"kubernetes.io/projected/aab63c87-1d86-4b61-ab5d-2bc8ad087853-kube-api-access-bz229\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:31.996768 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:31.996736    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"]
Jul 28 18:12:32.002093 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:32.002058    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6f2vr"]
Jul 28 18:12:33.180302 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:33.180258    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5790585c-ec62-43c7-8fc6-fe883aad84e6" path="/var/lib/kubelet/pods/5790585c-ec62-43c7-8fc6-fe883aad84e6/volumes"
Jul 28 18:12:35.022308 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.022265    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.028350 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.028320    2585 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.123108 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.123079    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.123234 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.123188    2585 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.126592 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.126571    2585 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") DevicePath \"csi-11cfe839c471910a057ef59d0eae056bd23fb2f5122acbf71037c2b463b10a31\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.335346 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.335311    2585 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d80372062c10f11126b77088bee6191fef8e1bbabe19894b05b047257cca869e/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.348620 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.348592    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"e2e-pvc-model-download-s2mpj\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.404884 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.404852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:35.533521 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.533495    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"]
Jul 28 18:12:35.535347 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:12:35.535322    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaab63c87_1d86_4b61_ab5d_2bc8ad087853.slice/crio-545f8bb2fdc088c3ee8abfec33273a0272128ac82883d6de80857ac09e684b72 WatchSource:0}: Error finding container 545f8bb2fdc088c3ee8abfec33273a0272128ac82883d6de80857ac09e684b72: Status 404 returned error can't find the container with id 545f8bb2fdc088c3ee8abfec33273a0272128ac82883d6de80857ac09e684b72
Jul 28 18:12:35.875900 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.875869    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj" event={"ID":"aab63c87-1d86-4b61-ab5d-2bc8ad087853","Type":"ContainerStarted","Data":"ece4115d31d24aa8aaaa1700ee42b7e721abfacbba1689cc56ac85e98f67aaab"}
Jul 28 18:12:35.876050 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.875909    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj" event={"ID":"aab63c87-1d86-4b61-ab5d-2bc8ad087853","Type":"ContainerStarted","Data":"545f8bb2fdc088c3ee8abfec33273a0272128ac82883d6de80857ac09e684b72"}
Jul 28 18:12:35.894635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:35.894580    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj" podStartSLOduration=8.89456573 podStartE2EDuration="8.89456573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:12:35.893095387 +0000 UTC m=+3341.236687664" watchObservedRunningTime="2026-07-28 18:12:35.89456573 +0000 UTC m=+3341.238157997"
Jul 28 18:12:44.916826 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:44.916790    2585 generic.go:356] "Generic (PLEG): container finished" podID="aab63c87-1d86-4b61-ab5d-2bc8ad087853" containerID="ece4115d31d24aa8aaaa1700ee42b7e721abfacbba1689cc56ac85e98f67aaab" exitCode=0
Jul 28 18:12:44.917241 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:44.916864    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj" event={"ID":"aab63c87-1d86-4b61-ab5d-2bc8ad087853","Type":"ContainerDied","Data":"ece4115d31d24aa8aaaa1700ee42b7e721abfacbba1689cc56ac85e98f67aaab"}
Jul 28 18:12:46.065453 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.065425    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:46.223912 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.223828    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") "
Jul 28 18:12:46.223912 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.223893    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz229\" (UniqueName: \"kubernetes.io/projected/aab63c87-1d86-4b61-ab5d-2bc8ad087853-kube-api-access-bz229\") pod \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\" (UID: \"aab63c87-1d86-4b61-ab5d-2bc8ad087853\") "
Jul 28 18:12:46.226236 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.226202    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab63c87-1d86-4b61-ab5d-2bc8ad087853-kube-api-access-bz229" (OuterVolumeSpecName: "kube-api-access-bz229") pod "aab63c87-1d86-4b61-ab5d-2bc8ad087853" (UID: "aab63c87-1d86-4b61-ab5d-2bc8ad087853"). InnerVolumeSpecName "kube-api-access-bz229". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:12:46.226827 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.226807    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b" (OuterVolumeSpecName: "model-storage") pod "aab63c87-1d86-4b61-ab5d-2bc8ad087853" (UID: "aab63c87-1d86-4b61-ab5d-2bc8ad087853"). InnerVolumeSpecName "pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:12:46.324450 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.324417    2585 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") on node \"ip-10-0-135-20.ec2.internal\" "
Jul 28 18:12:46.324450 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.324444    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz229\" (UniqueName: \"kubernetes.io/projected/aab63c87-1d86-4b61-ab5d-2bc8ad087853-kube-api-access-bz229\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:12:46.442972 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.442948    2585 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b") on node "ip-10-0-135-20.ec2.internal"
Jul 28 18:12:46.526392 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.526361    2585 reconciler_common.go:299] "Volume detached for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"csi-11cfe839c471910a057ef59d0eae056bd23fb2f5122acbf71037c2b463b10a31\""
Jul 28 18:12:46.927044 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.926956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj" event={"ID":"aab63c87-1d86-4b61-ab5d-2bc8ad087853","Type":"ContainerDied","Data":"545f8bb2fdc088c3ee8abfec33273a0272128ac82883d6de80857ac09e684b72"}
Jul 28 18:12:46.927044 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.926974    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj"
Jul 28 18:12:46.927044 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:12:46.926994    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545f8bb2fdc088c3ee8abfec33273a0272128ac82883d6de80857ac09e684b72"
Jul 28 18:13:01.340502 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.340468    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"]
Jul 28 18:13:01.341076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.340774    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab63c87-1d86-4b61-ab5d-2bc8ad087853" containerName="storage-initializer"
Jul 28 18:13:01.341076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.340787    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab63c87-1d86-4b61-ab5d-2bc8ad087853" containerName="storage-initializer"
Jul 28 18:13:01.341076 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.340842    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="aab63c87-1d86-4b61-ab5d-2bc8ad087853" containerName="storage-initializer"
Jul 28 18:13:01.345168 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.345146    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.348030 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.347985    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 18:13:01.348160 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.348076    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-w6gcf\""
Jul 28 18:13:01.348160 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.348088    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2wqp2\""
Jul 28 18:13:01.348251 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.348155    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 18:13:01.349131 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.349115    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 18:13:01.359442 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.359421    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"]
Jul 28 18:13:01.431870 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.431837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.432023 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.431883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.432023 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.431906    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50febc0e-e107-4f9e-9767-670777188d7c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.432023 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.431932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.432023 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.431954    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533021 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.532986    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533021 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533027    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50febc0e-e107-4f9e-9767-670777188d7c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533260 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533260 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533089    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwb6\" (UniqueName: \"kubernetes.io/projected/50febc0e-e107-4f9e-9767-670777188d7c-kube-api-access-mrwb6\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533260 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533120    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533406 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533448 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533414    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533493 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533466    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.533637 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.533620    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.535385 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.535360    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50febc0e-e107-4f9e-9767-670777188d7c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.535661 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.535643    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50febc0e-e107-4f9e-9767-670777188d7c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.572999 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.572963    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"]
Jul 28 18:13:01.576365 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.576344    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:01.579181 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.579161    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-82p9w\""
Jul 28 18:13:01.586393 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.586371    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"]
Jul 28 18:13:01.634651 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.634567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwb6\" (UniqueName: \"kubernetes.io/projected/50febc0e-e107-4f9e-9767-670777188d7c-kube-api-access-mrwb6\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.653104 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.653079    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwb6\" (UniqueName: \"kubernetes.io/projected/50febc0e-e107-4f9e-9767-670777188d7c-kube-api-access-mrwb6\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" (UID: \"50febc0e-e107-4f9e-9767-670777188d7c\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:13:01.735969 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.735923    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150a3f51-fd04-40ce-84ce-801888b42c71-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66\" (UID: \"150a3f51-fd04-40ce-84ce-801888b42c71\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:01.735969 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.735973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bqn\" (UniqueName: \"kubernetes.io/projected/150a3f51-fd04-40ce-84ce-801888b42c71-kube-api-access-s7bqn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66\" (UID: \"150a3f51-fd04-40ce-84ce-801888b42c71\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:01.836835 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.836778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150a3f51-fd04-40ce-84ce-801888b42c71-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66\" (UID: \"150a3f51-fd04-40ce-84ce-801888b42c71\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:01.837042 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.836844    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bqn\" (UniqueName: \"kubernetes.io/projected/150a3f51-fd04-40ce-84ce-801888b42c71-kube-api-access-s7bqn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66\" (UID: \"150a3f51-fd04-40ce-84ce-801888b42c71\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:01.839280 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.839245    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150a3f51-fd04-40ce-84ce-801888b42c71-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66\" (UID: \"150a3f51-fd04-40ce-84ce-801888b42c71\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:01.847067 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.847040    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bqn\" (UniqueName: \"kubernetes.io/projected/150a3f51-fd04-40ce-84ce-801888b42c71-kube-api-access-s7bqn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66\" (UID: \"150a3f51-fd04-40ce-84ce-801888b42c71\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:01.887720 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:01.887637    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:02.018154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:02.018127    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"]
Jul 28 18:13:02.019701 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:13:02.019667    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150a3f51_fd04_40ce_84ce_801888b42c71.slice/crio-2930fd757e81d82bd669c1919beea2aabe4b0fcf5b8a548a4fea1ea9b90b6bc8 WatchSource:0}: Error finding container 2930fd757e81d82bd669c1919beea2aabe4b0fcf5b8a548a4fea1ea9b90b6bc8: Status 404 returned error can't find the container with id 2930fd757e81d82bd669c1919beea2aabe4b0fcf5b8a548a4fea1ea9b90b6bc8
Jul 28 18:13:02.994026 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:02.993990    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66" event={"ID":"150a3f51-fd04-40ce-84ce-801888b42c71","Type":"ContainerStarted","Data":"6cf5e48b4e4fbf44d539e6220f03fc5333bb7d66fa5bd7584b6ef4d9d796c712"}
Jul 28 18:13:02.994440 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:02.994051    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66" event={"ID":"150a3f51-fd04-40ce-84ce-801888b42c71","Type":"ContainerStarted","Data":"2930fd757e81d82bd669c1919beea2aabe4b0fcf5b8a548a4fea1ea9b90b6bc8"}
Jul 28 18:13:02.994440 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:02.994082    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:13:03.012335 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:03.012289    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66" podStartSLOduration=2.012274828 podStartE2EDuration="2.012274828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:13:03.0118392 +0000 UTC m=+3368.355431478" watchObservedRunningTime="2026-07-28 18:13:03.012274828 +0000 UTC m=+3368.355867105"
Jul 28 18:13:33.999920 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:13:33.999888    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66"
Jul 28 18:15:04.354450 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:15:04.354405    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:15:04.471774 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:15:04.471743    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:15:55.489936 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:15:55.489904    2585 scope.go:117] "RemoveContainer" containerID="357745ffe1cc3c8bf7ebb3067e89471d03931d4141b5840afcb35679a2d9b1ae"
Jul 28 18:16:55.293623 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:16:55.293496    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:16:55.304616 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:16:55.304595    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:17:07.473098 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:17:07.473044    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:17:07.954611 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:17:07.954580    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:19:10.955442 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:19:10.955398    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:19:11.426708 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:19:11.426676    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:21:14.427622 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:21:14.427581    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:21:14.899624 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:21:14.899593    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:21:55.320543 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:21:55.320437    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:21:55.336609 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:21:55.336584    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:23:17.900700 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:23:17.900612    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:23:18.365590 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:23:18.365542    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:25:21.366980 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:25:21.366940    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:25:21.852608 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:25:21.852577    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:26:55.343916 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:26:55.343813    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:26:55.361276 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:26:55.361255    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:27:24.853827 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:27:24.853731    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:27:25.319584 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:27:25.319531    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:27:48.431611 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:27:48.431574    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66_150a3f51-fd04-40ce-84ce-801888b42c71/main/0.log"
Jul 28 18:28:46.931375 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.931284    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:28:46.934536 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.934517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:46.946469 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.946446    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-vzspf\""
Jul 28 18:28:46.946469 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.946455    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s6n7w\""
Jul 28 18:28:46.946667 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.946456    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 18:28:46.946667 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.946447    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 18:28:46.946667 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.946466    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 18:28:46.959334 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:46.959312    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:28:47.065747 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.065709    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.065955 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.065754    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.065955 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.065791    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.065955 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.065842    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbsj\" (UniqueName: \"kubernetes.io/projected/6436baeb-45ff-4c36-8872-c35edb5c3708-kube-api-access-zwbsj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.065955 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.065886    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.065955 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.065928    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6436baeb-45ff-4c36-8872-c35edb5c3708-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167058 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167016    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbsj\" (UniqueName: \"kubernetes.io/projected/6436baeb-45ff-4c36-8872-c35edb5c3708-kube-api-access-zwbsj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167266 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167075    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167266 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6436baeb-45ff-4c36-8872-c35edb5c3708-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167266 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167137    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167266 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167161    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167266 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167516 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167618 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167604    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.167687 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.167627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.169397 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.169378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.169629 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.169612    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6436baeb-45ff-4c36-8872-c35edb5c3708-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:47.184461 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:47.184399    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbsj\" (UniqueName: \"kubernetes.io/projected/6436baeb-45ff-4c36-8872-c35edb5c3708-kube-api-access-zwbsj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.241751 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.241719    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.248410 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.248379    2585 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.342667 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.342625    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.342856 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.342741    2585 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.346325 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.346299    2585 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") DevicePath \"csi-04a352be974307a729b89cef009a9ceec49d85a0c436e560c3bcd7961e96486c\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.434394 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.434357    2585 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a80a8c5ede8dcfc70075ab4f15cd30d508cac99bea6759721e264a2a2aabee9a/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.447264 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.447231    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.648927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.648890    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-vzspf\""
Jul 28 18:28:55.658757 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.658732    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s6n7w\""
Jul 28 18:28:55.666472 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.666454    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:28:55.804440 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:55.804411    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:28:55.806511 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:28:55.806484    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6436baeb_45ff_4c36_8872_c35edb5c3708.slice/crio-2729e0982deb088febc9ce9cb4264030b4350010842d7f92b0a4d2604f348d67 WatchSource:0}: Error finding container 2729e0982deb088febc9ce9cb4264030b4350010842d7f92b0a4d2604f348d67: Status 404 returned error can't find the container with id 2729e0982deb088febc9ce9cb4264030b4350010842d7f92b0a4d2604f348d67
Jul 28 18:28:56.677401 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:56.677359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6436baeb-45ff-4c36-8872-c35edb5c3708","Type":"ContainerStarted","Data":"507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f"}
Jul 28 18:28:56.677809 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:56.677458    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6436baeb-45ff-4c36-8872-c35edb5c3708","Type":"ContainerStarted","Data":"2729e0982deb088febc9ce9cb4264030b4350010842d7f92b0a4d2604f348d67"}
Jul 28 18:28:56.699919 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:56.699858    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=10.699840867 podStartE2EDuration="10.699840867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:56.698269789 +0000 UTC m=+4322.041862093" watchObservedRunningTime="2026-07-28 18:28:56.699840867 +0000 UTC m=+4322.043433144"
Jul 28 18:28:57.666803 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:28:57.666763    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:29:05.667184 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:05.667143    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:29:05.668394 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:05.668363    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:29:15.666880 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:15.666839    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:29:25.667867 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:25.667823    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:29:28.320975 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:29:28.320923    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:29:28.832946 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:28.832913    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:29:35.667257 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:35.667204    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:29:45.667397 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:45.667355    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:29:55.667895 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:29:55.667849    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:30:05.667184 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:05.667136    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:30:15.667502 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:15.667446    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:30:25.667173 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:25.667065    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 18:30:35.676864 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:35.676826    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:30:35.684449 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:35.684421    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:30:42.887757 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:42.887721    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:30:57.909916 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:57.909851    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main" containerID="cri-o://507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f" gracePeriod=45
Jul 28 18:30:58.653507 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.653480    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:30:58.763097 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763016    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-home\") pod \"6436baeb-45ff-4c36-8872-c35edb5c3708\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") "
Jul 28 18:30:58.763097 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763058    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-model-cache\") pod \"6436baeb-45ff-4c36-8872-c35edb5c3708\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") "
Jul 28 18:30:58.763097 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763089    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-tmp-dir\") pod \"6436baeb-45ff-4c36-8872-c35edb5c3708\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") "
Jul 28 18:30:58.763335 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763183    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") pod \"6436baeb-45ff-4c36-8872-c35edb5c3708\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") "
Jul 28 18:30:58.763335 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763271    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwbsj\" (UniqueName: \"kubernetes.io/projected/6436baeb-45ff-4c36-8872-c35edb5c3708-kube-api-access-zwbsj\") pod \"6436baeb-45ff-4c36-8872-c35edb5c3708\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") "
Jul 28 18:30:58.763335 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763310    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-dshm\") pod \"6436baeb-45ff-4c36-8872-c35edb5c3708\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") "
Jul 28 18:30:58.763471 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763356    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6436baeb-45ff-4c36-8872-c35edb5c3708-tls-certs\") pod \"6436baeb-45ff-4c36-8872-c35edb5c3708\" (UID: \"6436baeb-45ff-4c36-8872-c35edb5c3708\") "
Jul 28 18:30:58.763471 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763305    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-model-cache" (OuterVolumeSpecName: "model-cache") pod "6436baeb-45ff-4c36-8872-c35edb5c3708" (UID: "6436baeb-45ff-4c36-8872-c35edb5c3708"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:58.763709 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763655    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.763709 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.763692    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-home" (OuterVolumeSpecName: "home") pod "6436baeb-45ff-4c36-8872-c35edb5c3708" (UID: "6436baeb-45ff-4c36-8872-c35edb5c3708"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:58.765925 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.765887    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-dshm" (OuterVolumeSpecName: "dshm") pod "6436baeb-45ff-4c36-8872-c35edb5c3708" (UID: "6436baeb-45ff-4c36-8872-c35edb5c3708"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:58.766130 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.766107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436baeb-45ff-4c36-8872-c35edb5c3708-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6436baeb-45ff-4c36-8872-c35edb5c3708" (UID: "6436baeb-45ff-4c36-8872-c35edb5c3708"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:58.766224 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.766111    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3" (OuterVolumeSpecName: "kserve-pvc-source") pod "6436baeb-45ff-4c36-8872-c35edb5c3708" (UID: "6436baeb-45ff-4c36-8872-c35edb5c3708"). InnerVolumeSpecName "pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:30:58.766224 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.766128    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6436baeb-45ff-4c36-8872-c35edb5c3708-kube-api-access-zwbsj" (OuterVolumeSpecName: "kube-api-access-zwbsj") pod "6436baeb-45ff-4c36-8872-c35edb5c3708" (UID: "6436baeb-45ff-4c36-8872-c35edb5c3708"). InnerVolumeSpecName "kube-api-access-zwbsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:58.778191 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.778163    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6436baeb-45ff-4c36-8872-c35edb5c3708" (UID: "6436baeb-45ff-4c36-8872-c35edb5c3708"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:58.864490 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.864457    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.864490 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.864485    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.864700 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.864512    2585 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") on node \"ip-10-0-135-20.ec2.internal\" "
Jul 28 18:30:58.864700 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.864523    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbsj\" (UniqueName: \"kubernetes.io/projected/6436baeb-45ff-4c36-8872-c35edb5c3708-kube-api-access-zwbsj\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.864700 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.864531    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6436baeb-45ff-4c36-8872-c35edb5c3708-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.864700 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.864540    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6436baeb-45ff-4c36-8872-c35edb5c3708-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.892274 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.892255    2585 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3") on node "ip-10-0-135-20.ec2.internal"
Jul 28 18:30:58.965571 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:58.965521    2585 reconciler_common.go:299] "Volume detached for volume \"pvc-322c2b42-7918-4d3d-a8b0-0cdf0543dfba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0adc5b422695948e3\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"csi-04a352be974307a729b89cef009a9ceec49d85a0c436e560c3bcd7961e96486c\""
Jul 28 18:30:59.248927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.248875    2585 generic.go:356] "Generic (PLEG): container finished" podID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerID="507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f" exitCode=0
Jul 28 18:30:59.248927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.248923    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:30:59.249166 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.248968    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6436baeb-45ff-4c36-8872-c35edb5c3708","Type":"ContainerDied","Data":"507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f"}
Jul 28 18:30:59.249166 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.249038    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6436baeb-45ff-4c36-8872-c35edb5c3708","Type":"ContainerDied","Data":"2729e0982deb088febc9ce9cb4264030b4350010842d7f92b0a4d2604f348d67"}
Jul 28 18:30:59.249166 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.249074    2585 scope.go:117] "RemoveContainer" containerID="507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f"
Jul 28 18:30:59.259523 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.259499    2585 scope.go:117] "RemoveContainer" containerID="507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f"
Jul 28 18:30:59.259817 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:30:59.259793    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f\": container with ID starting with 507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f not found: ID does not exist" containerID="507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f"
Jul 28 18:30:59.259941 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.259826    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f"} err="failed to get container status \"507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f\": rpc error: code = NotFound desc = could not find container \"507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f\": container with ID starting with 507ba33f778a53ceb863c401d60fc0b8d84ab0a89a82e28782afb3e234a9a02f not found: ID does not exist"
Jul 28 18:30:59.270344 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.270317    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:30:59.274476 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:30:59.274456    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:31:01.179738 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:01.179702    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" path="/var/lib/kubelet/pods/6436baeb-45ff-4c36-8872-c35edb5c3708/volumes"
Jul 28 18:31:31.834442 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:31:31.834396    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:31:32.390036 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:32.390002    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:31:47.821661 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.821575    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"]
Jul 28 18:31:47.822108 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.821908    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main"
Jul 28 18:31:47.822108 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.821921    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main"
Jul 28 18:31:47.822108 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.821983    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6436baeb-45ff-4c36-8872-c35edb5c3708" containerName="main"
Jul 28 18:31:47.825141 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.825121    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:47.828060 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.828033    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 18:31:47.828185 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.828097    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 18:31:47.828185 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.828137    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 18:31:47.828185 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.828147    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-vtm2f\""
Jul 28 18:31:47.828185 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.828151    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 18:31:47.836446 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.836426    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"]
Jul 28 18:31:47.973645 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.973609    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-dshm\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:47.973645 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.973648    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tls-certs\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:47.973854 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.973747    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-model-cache\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:47.973854 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.973790    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kserve-provision-location\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:47.973854 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.973825    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-home\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:47.973960 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.973859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gjr\" (UniqueName: \"kubernetes.io/projected/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kube-api-access-m6gjr\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:47.973960 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:47.973882    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tmp-dir\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075350 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-dshm\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075350 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075303    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tls-certs\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075569 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075351    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-model-cache\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075569 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075388    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kserve-provision-location\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075569 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075529    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-home\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075753 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075612    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gjr\" (UniqueName: \"kubernetes.io/projected/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kube-api-access-m6gjr\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075753 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075649    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tmp-dir\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075805    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kserve-provision-location\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075872 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075852    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-model-cache\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.075987 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-home\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.076046 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.075995    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tmp-dir\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.077571 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.077534    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-dshm\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.077957 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.077937    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tls-certs\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.085394 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.085372    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gjr\" (UniqueName: \"kubernetes.io/projected/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kube-api-access-m6gjr\") pod \"stop-feature-test-kserve-5c6b8c586b-8dvhs\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.135010 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.134987    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:48.269120 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.269096    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"]
Jul 28 18:31:48.271477 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:31:48.271440    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af8746f_d0b8_4ada_b6c9_4c4cd2a35ca8.slice/crio-feb63fdc36e6ee708b77068f93bac3e53834a489f3454012590f71452230cdab WatchSource:0}: Error finding container feb63fdc36e6ee708b77068f93bac3e53834a489f3454012590f71452230cdab: Status 404 returned error can't find the container with id feb63fdc36e6ee708b77068f93bac3e53834a489f3454012590f71452230cdab
Jul 28 18:31:48.460630 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.460499    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" event={"ID":"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8","Type":"ContainerStarted","Data":"632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004"}
Jul 28 18:31:48.460630 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:48.460587    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" event={"ID":"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8","Type":"ContainerStarted","Data":"feb63fdc36e6ee708b77068f93bac3e53834a489f3454012590f71452230cdab"}
Jul 28 18:31:55.368356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:55.368208    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:31:55.386095 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:55.386069    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:31:55.496403 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:55.496365    2585 generic.go:356] "Generic (PLEG): container finished" podID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerID="632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004" exitCode=0
Jul 28 18:31:55.496593 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:55.496446    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" event={"ID":"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8","Type":"ContainerDied","Data":"632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004"}
Jul 28 18:31:56.503262 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:56.503221    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" event={"ID":"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8","Type":"ContainerStarted","Data":"17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a"}
Jul 28 18:31:56.529228 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:56.529157    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podStartSLOduration=2.785923309 podStartE2EDuration="9.529133907s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.743210598s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:56.525302916 +0000 UTC m=+4501.868895205" watchObservedRunningTime="2026-07-28 18:31:56.529133907 +0000 UTC m=+4501.872726185"
Jul 28 18:31:57.135715 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:57.135614    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:58.135632 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:58.135523    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:31:58.137451 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:31:58.137411    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:32:08.135971 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:32:08.135919    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:32:18.135903 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:32:18.135861    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:32:28.135739 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:32:28.135692    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:32:38.136382 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:32:38.136322    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:32:48.136235 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:32:48.136192    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:32:58.136049 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:32:58.136005    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:33:08.135447 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:33:08.135399    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:33:18.135764 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:33:18.135669    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:33:28.136192 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:33:28.136149    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:33:35.391641 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:33:35.391583    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:33:35.947474 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:33:35.947432    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:33:38.135451 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:33:38.135409    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:33:48.145773 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:33:48.145737    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:33:48.153511 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:33:48.153485    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:34:05.507812 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:05.507773    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"]
Jul 28 18:34:20.530923 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:20.530843    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" containerID="cri-o://17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a" gracePeriod=45
Jul 28 18:34:21.147095 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.146939    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 18:34:21.286023 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.285999    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:34:21.380123 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380041    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tls-certs\") pod \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") "
Jul 28 18:34:21.380123 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380089    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-home\") pod \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") "
Jul 28 18:34:21.380345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380142    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-model-cache\") pod \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") "
Jul 28 18:34:21.380345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380160    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tmp-dir\") pod \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") "
Jul 28 18:34:21.380345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380185    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kserve-provision-location\") pod \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") "
Jul 28 18:34:21.380345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380212    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6gjr\" (UniqueName: \"kubernetes.io/projected/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kube-api-access-m6gjr\") pod \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") "
Jul 28 18:34:21.380345 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380245    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-dshm\") pod \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\" (UID: \"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8\") "
Jul 28 18:34:21.380635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.380441    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-model-cache" (OuterVolumeSpecName: "model-cache") pod "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" (UID: "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:21.381179 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.381151    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-home" (OuterVolumeSpecName: "home") pod "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" (UID: "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:21.383101 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.383073    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kube-api-access-m6gjr" (OuterVolumeSpecName: "kube-api-access-m6gjr") pod "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" (UID: "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8"). InnerVolumeSpecName "kube-api-access-m6gjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:21.383240 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.383069    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" (UID: "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:21.383240 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.383150    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-dshm" (OuterVolumeSpecName: "dshm") pod "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" (UID: "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:21.392264 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.392238    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" (UID: "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:21.452866 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.452803    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" (UID: "1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:21.481717 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.481681    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.481717 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.481707    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.481717 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.481718    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.482007 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.481727    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.482007 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.481736    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gjr\" (UniqueName: \"kubernetes.io/projected/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-kube-api-access-m6gjr\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.482007 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.481744    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.482007 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:21.481752    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:34:22.140888 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.140849    2585 generic.go:356] "Generic (PLEG): container finished" podID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerID="17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a" exitCode=0
Jul 28 18:34:22.141329 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.140919    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"
Jul 28 18:34:22.141329 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.140935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" event={"ID":"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8","Type":"ContainerDied","Data":"17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a"}
Jul 28 18:34:22.141329 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.141015    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs" event={"ID":"1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8","Type":"ContainerDied","Data":"feb63fdc36e6ee708b77068f93bac3e53834a489f3454012590f71452230cdab"}
Jul 28 18:34:22.141329 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.141038    2585 scope.go:117] "RemoveContainer" containerID="17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a"
Jul 28 18:34:22.149710 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.149690    2585 scope.go:117] "RemoveContainer" containerID="632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004"
Jul 28 18:34:22.157450 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.157430    2585 scope.go:117] "RemoveContainer" containerID="17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a"
Jul 28 18:34:22.157706 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:34:22.157686    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a\": container with ID starting with 17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a not found: ID does not exist" containerID="17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a"
Jul 28 18:34:22.157754 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.157715    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a"} err="failed to get container status \"17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a\": rpc error: code = NotFound desc = could not find container \"17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a\": container with ID starting with 17f30b1f20d3d44ac09408812597bc5378cf14b5e3935390a9f9f89bb8cb2d6a not found: ID does not exist"
Jul 28 18:34:22.157754 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.157730    2585 scope.go:117] "RemoveContainer" containerID="632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004"
Jul 28 18:34:22.157954 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:34:22.157936    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004\": container with ID starting with 632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004 not found: ID does not exist" containerID="632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004"
Jul 28 18:34:22.157989 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.157961    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004"} err="failed to get container status \"632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004\": rpc error: code = NotFound desc = could not find container \"632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004\": container with ID starting with 632b538a276343e1ae5badb9d45c4199c367ce9bbfdfab8374fe343dda819004 not found: ID does not exist"
Jul 28 18:34:22.163122 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.163102    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"]
Jul 28 18:34:22.167514 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:22.167494    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-8dvhs"]
Jul 28 18:34:23.180717 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:23.180689    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" path="/var/lib/kubelet/pods/1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8/volumes"
Jul 28 18:34:47.543189 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.543094    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"]
Jul 28 18:34:47.543625 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.543455    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main"
Jul 28 18:34:47.543625 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.543469    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main"
Jul 28 18:34:47.543625 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.543491    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="storage-initializer"
Jul 28 18:34:47.543625 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.543497    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="storage-initializer"
Jul 28 18:34:47.543625 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.543542    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1af8746f-d0b8-4ada-b6c9-4c4cd2a35ca8" containerName="main"
Jul 28 18:34:47.545403 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.545375    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.551720 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.551687    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 18:34:47.551869 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.551703    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-vtm2f\""
Jul 28 18:34:47.551869 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.551854    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 18:34:47.552142 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.552124    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 18:34:47.552695 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.552676    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 18:34:47.560375 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.560351    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"]
Jul 28 18:34:47.600669 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.600640    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-model-cache\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.600818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.600672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-tmp-dir\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.600818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.600693    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444cd\" (UniqueName: \"kubernetes.io/projected/b808455b-7668-4b40-8759-b3c24a173c24-kube-api-access-444cd\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.600818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.600716    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-dshm\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.600818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.600734    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b808455b-7668-4b40-8759-b3c24a173c24-tls-certs\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.600818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.600795    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-home\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.600818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.600813    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-kserve-provision-location\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.701679 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.701644    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-dshm\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.701679 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.701681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b808455b-7668-4b40-8759-b3c24a173c24-tls-certs\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.701927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.701741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-home\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.701927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.701760    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-kserve-provision-location\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.701927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.701798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-model-cache\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.701927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.701824    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-tmp-dir\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.701927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.701847    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444cd\" (UniqueName: \"kubernetes.io/projected/b808455b-7668-4b40-8759-b3c24a173c24-kube-api-access-444cd\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.702267 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.702202    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-kserve-provision-location\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.702267 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.702218    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-home\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.702365 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.702301    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-model-cache\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.702448 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.702431    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-tmp-dir\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.703930 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.703907    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-dshm\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.704278 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.704261    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b808455b-7668-4b40-8759-b3c24a173c24-tls-certs\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.710752 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.710729    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-444cd\" (UniqueName: \"kubernetes.io/projected/b808455b-7668-4b40-8759-b3c24a173c24-kube-api-access-444cd\") pod \"stop-feature-test-kserve-5c6b8c586b-bk8cj\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.861719 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.861625    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:47.998377 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:47.998338    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"]
Jul 28 18:34:48.000698 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:34:48.000661    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb808455b_7668_4b40_8759_b3c24a173c24.slice/crio-0ee498512bd6a899482bb8b24a3b3039d9336c62dbe3c9b51155106ea5f79bcf WatchSource:0}: Error finding container 0ee498512bd6a899482bb8b24a3b3039d9336c62dbe3c9b51155106ea5f79bcf: Status 404 returned error can't find the container with id 0ee498512bd6a899482bb8b24a3b3039d9336c62dbe3c9b51155106ea5f79bcf
Jul 28 18:34:48.246646 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:48.246544    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" event={"ID":"b808455b-7668-4b40-8759-b3c24a173c24","Type":"ContainerStarted","Data":"ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48"}
Jul 28 18:34:48.246646 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:48.246609    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" event={"ID":"b808455b-7668-4b40-8759-b3c24a173c24","Type":"ContainerStarted","Data":"0ee498512bd6a899482bb8b24a3b3039d9336c62dbe3c9b51155106ea5f79bcf"}
Jul 28 18:34:53.270134 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:53.270099    2585 generic.go:356] "Generic (PLEG): container finished" podID="b808455b-7668-4b40-8759-b3c24a173c24" containerID="ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48" exitCode=0
Jul 28 18:34:53.270134 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:53.270135    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" event={"ID":"b808455b-7668-4b40-8759-b3c24a173c24","Type":"ContainerDied","Data":"ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48"}
Jul 28 18:34:54.277293 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:54.277257    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" event={"ID":"b808455b-7668-4b40-8759-b3c24a173c24","Type":"ContainerStarted","Data":"70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef"}
Jul 28 18:34:54.302150 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:54.302100    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podStartSLOduration=2.359654965 podStartE2EDuration="7.302085763s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.942430798s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:54.300207849 +0000 UTC m=+4679.643800125" watchObservedRunningTime="2026-07-28 18:34:54.302085763 +0000 UTC m=+4679.645678040"
Jul 28 18:34:54.862069 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:54.862026    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:57.862192 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:57.862154    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:34:57.863478 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:34:57.863448    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:35:07.862292 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:35:07.862230    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:35:17.863279 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:35:17.863209    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:35:27.863012 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:35:27.862967    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:35:37.863046 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:35:37.863002    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:35:38.948166 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:35:38.948122    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:35:39.481342 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:35:39.481307    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:35:47.862714 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:35:47.862668    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:35:57.862297 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:35:57.862239    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:36:07.862519 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:07.862462    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:36:17.862590 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:17.862458    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:36:27.862386 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:27.862340    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:36:37.862293 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:37.862247    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:36:47.872627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:47.872593    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:36:47.880914 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:47.880890    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:36:55.392708 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:55.392581    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:36:55.411412 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:55.411387    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:36:59.270924 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:36:59.270888    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"]
Jul 28 18:37:14.293069 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:14.293021    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" containerID="cri-o://70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef" gracePeriod=45
Jul 28 18:37:14.874018 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:14.873972    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 18:37:15.041591 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.041567    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:37:15.219073 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.218998    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-dshm\") pod \"b808455b-7668-4b40-8759-b3c24a173c24\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") "
Jul 28 18:37:15.219073 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219047    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-tmp-dir\") pod \"b808455b-7668-4b40-8759-b3c24a173c24\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") "
Jul 28 18:37:15.219073 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219073    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-444cd\" (UniqueName: \"kubernetes.io/projected/b808455b-7668-4b40-8759-b3c24a173c24-kube-api-access-444cd\") pod \"b808455b-7668-4b40-8759-b3c24a173c24\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") "
Jul 28 18:37:15.219373 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219090    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-home\") pod \"b808455b-7668-4b40-8759-b3c24a173c24\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") "
Jul 28 18:37:15.219373 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219131    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-kserve-provision-location\") pod \"b808455b-7668-4b40-8759-b3c24a173c24\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") "
Jul 28 18:37:15.219373 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219205    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-model-cache\") pod \"b808455b-7668-4b40-8759-b3c24a173c24\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") "
Jul 28 18:37:15.219373 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219260    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b808455b-7668-4b40-8759-b3c24a173c24-tls-certs\") pod \"b808455b-7668-4b40-8759-b3c24a173c24\" (UID: \"b808455b-7668-4b40-8759-b3c24a173c24\") "
Jul 28 18:37:15.219611 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219515    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-model-cache" (OuterVolumeSpecName: "model-cache") pod "b808455b-7668-4b40-8759-b3c24a173c24" (UID: "b808455b-7668-4b40-8759-b3c24a173c24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:15.219836 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219801    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-home" (OuterVolumeSpecName: "home") pod "b808455b-7668-4b40-8759-b3c24a173c24" (UID: "b808455b-7668-4b40-8759-b3c24a173c24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:15.219924 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219882    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.219924 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.219902    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.221459 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.221430    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-dshm" (OuterVolumeSpecName: "dshm") pod "b808455b-7668-4b40-8759-b3c24a173c24" (UID: "b808455b-7668-4b40-8759-b3c24a173c24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:15.221865 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.221829    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b808455b-7668-4b40-8759-b3c24a173c24-kube-api-access-444cd" (OuterVolumeSpecName: "kube-api-access-444cd") pod "b808455b-7668-4b40-8759-b3c24a173c24" (UID: "b808455b-7668-4b40-8759-b3c24a173c24"). InnerVolumeSpecName "kube-api-access-444cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:37:15.222110 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.222080    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b808455b-7668-4b40-8759-b3c24a173c24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b808455b-7668-4b40-8759-b3c24a173c24" (UID: "b808455b-7668-4b40-8759-b3c24a173c24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:37:15.231303 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.231281    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b808455b-7668-4b40-8759-b3c24a173c24" (UID: "b808455b-7668-4b40-8759-b3c24a173c24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:15.283676 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.283635    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b808455b-7668-4b40-8759-b3c24a173c24" (UID: "b808455b-7668-4b40-8759-b3c24a173c24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:15.320657 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.320628    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b808455b-7668-4b40-8759-b3c24a173c24-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.320657 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.320649    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.320657 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.320659    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.321169 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.320669    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-444cd\" (UniqueName: \"kubernetes.io/projected/b808455b-7668-4b40-8759-b3c24a173c24-kube-api-access-444cd\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.321169 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.320678    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b808455b-7668-4b40-8759-b3c24a173c24-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.890431 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.890392    2585 generic.go:356] "Generic (PLEG): container finished" podID="b808455b-7668-4b40-8759-b3c24a173c24" containerID="70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef" exitCode=0
Jul 28 18:37:15.890658 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.890433    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" event={"ID":"b808455b-7668-4b40-8759-b3c24a173c24","Type":"ContainerDied","Data":"70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef"}
Jul 28 18:37:15.890658 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.890458    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"
Jul 28 18:37:15.890658 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.890513    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj" event={"ID":"b808455b-7668-4b40-8759-b3c24a173c24","Type":"ContainerDied","Data":"0ee498512bd6a899482bb8b24a3b3039d9336c62dbe3c9b51155106ea5f79bcf"}
Jul 28 18:37:15.890658 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.890536    2585 scope.go:117] "RemoveContainer" containerID="70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef"
Jul 28 18:37:15.899535 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.899520    2585 scope.go:117] "RemoveContainer" containerID="ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48"
Jul 28 18:37:15.907255 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.907235    2585 scope.go:117] "RemoveContainer" containerID="70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef"
Jul 28 18:37:15.907483 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:37:15.907465    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef\": container with ID starting with 70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef not found: ID does not exist" containerID="70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef"
Jul 28 18:37:15.907527 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.907492    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef"} err="failed to get container status \"70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef\": rpc error: code = NotFound desc = could not find container \"70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef\": container with ID starting with 70e1dadc9b86fc4e365825d643a7a1120279d20a6144ed9febcc16b6923e19ef not found: ID does not exist"
Jul 28 18:37:15.907527 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.907508    2585 scope.go:117] "RemoveContainer" containerID="ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48"
Jul 28 18:37:15.907757 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:37:15.907735    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48\": container with ID starting with ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48 not found: ID does not exist" containerID="ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48"
Jul 28 18:37:15.907819 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.907765    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48"} err="failed to get container status \"ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48\": rpc error: code = NotFound desc = could not find container \"ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48\": container with ID starting with ac917eeed094485475088ba6319e63ee03fad719f61d0441b3e3ae91fd7eab48 not found: ID does not exist"
Jul 28 18:37:15.913564 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.913527    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"]
Jul 28 18:37:15.919016 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:15.918996    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5c6b8c586b-bk8cj"]
Jul 28 18:37:17.179701 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:17.179666    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b808455b-7668-4b40-8759-b3c24a173c24" path="/var/lib/kubelet/pods/b808455b-7668-4b40-8759-b3c24a173c24/volumes"
Jul 28 18:37:31.711727 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.711690    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"]
Jul 28 18:37:31.712209 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.711996    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main"
Jul 28 18:37:31.712209 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.712007    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main"
Jul 28 18:37:31.712209 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.712021    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="storage-initializer"
Jul 28 18:37:31.712209 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.712027    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="storage-initializer"
Jul 28 18:37:31.712209 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.712083    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b808455b-7668-4b40-8759-b3c24a173c24" containerName="main"
Jul 28 18:37:31.715015 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.714999    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.718222 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.718185    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 28 18:37:31.718222 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.718212    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-b7rth\""
Jul 28 18:37:31.718400 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.718240    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:37:31.718400 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.718258    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 28 18:37:31.718400 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.718261    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 28 18:37:31.718600 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.718585    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 28 18:37:31.741566 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.741527    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"]
Jul 28 18:37:31.753087 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.753202 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417b1d82-cebe-475f-9cce-c6bf2ac230fd-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.753202 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753115    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-home\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.753202 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753135    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tls-certs\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.753330 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753215    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2n4\" (UniqueName: \"kubernetes.io/projected/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kube-api-access-ff2n4\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.753330 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753283    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-model-cache\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.753408 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753337    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tmp-dir\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.753452 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.753402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-dshm\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.854565 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-dshm\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.854746 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.854864 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854742    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417b1d82-cebe-475f-9cce-c6bf2ac230fd-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.854864 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854776    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-home\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.854864 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854820    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tls-certs\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.854864 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854851    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2n4\" (UniqueName: \"kubernetes.io/projected/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kube-api-access-ff2n4\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.855128 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854909    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-model-cache\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.855128 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.854955    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tmp-dir\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.855128 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.855055    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.855291 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.855262    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-home\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.855291 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.855267    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-model-cache\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.855409 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.855292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tmp-dir\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.855825 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.855804    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417b1d82-cebe-475f-9cce-c6bf2ac230fd-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.857372 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.857338    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-dshm\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.857687 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.857668    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tls-certs\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:31.865889 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:31.865870    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2n4\" (UniqueName: \"kubernetes.io/projected/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kube-api-access-ff2n4\") pod \"lora-single-adapter-test-kserve-5b5cc56b97-dgbdg\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:32.025479 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:32.025444    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:32.160914 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:32.160849    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"]
Jul 28 18:37:32.163227 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:37:32.163197    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b1d82_cebe_475f_9cce_c6bf2ac230fd.slice/crio-1dcf772d4ced6846e8015754b6146ae5d017b0a1abccccae330707f319a26e3d WatchSource:0}: Error finding container 1dcf772d4ced6846e8015754b6146ae5d017b0a1abccccae330707f319a26e3d: Status 404 returned error can't find the container with id 1dcf772d4ced6846e8015754b6146ae5d017b0a1abccccae330707f319a26e3d
Jul 28 18:37:32.962291 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:32.962250    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" event={"ID":"417b1d82-cebe-475f-9cce-c6bf2ac230fd","Type":"ContainerStarted","Data":"6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d"}
Jul 28 18:37:32.962701 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:32.962311    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" event={"ID":"417b1d82-cebe-475f-9cce-c6bf2ac230fd","Type":"ContainerStarted","Data":"1dcf772d4ced6846e8015754b6146ae5d017b0a1abccccae330707f319a26e3d"}
Jul 28 18:37:38.991190 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:38.991151    2585 generic.go:356] "Generic (PLEG): container finished" podID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerID="6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d" exitCode=0
Jul 28 18:37:38.991190 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:38.991193    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" event={"ID":"417b1d82-cebe-475f-9cce-c6bf2ac230fd","Type":"ContainerDied","Data":"6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d"}
Jul 28 18:37:40.000635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:40.000590    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" event={"ID":"417b1d82-cebe-475f-9cce-c6bf2ac230fd","Type":"ContainerStarted","Data":"c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa"}
Jul 28 18:37:40.026065 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:40.026003    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podStartSLOduration=2.923451305 podStartE2EDuration="9.025986006s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.102534701s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:37:40.023826512 +0000 UTC m=+4845.367418792" watchObservedRunningTime="2026-07-28 18:37:40.025986006 +0000 UTC m=+4845.369578283"
Jul 28 18:37:40.026065 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:40.026058    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:42.025987 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:42.025941    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:37:42.027881 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:42.027837    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:37:42.482923 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:37:42.482800    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:37:43.017817 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:43.017783    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:37:52.026100 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:37:52.025996    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:38:02.026262 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:38:02.026208    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:38:12.026321 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:38:12.026274    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:38:22.026196 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:38:22.026135    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:38:32.026026 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:38:32.025970    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:38:42.026757 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:38:42.026704    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:38:52.026470 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:38:52.026418    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:39:02.035732 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:39:02.035698    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:39:02.043462 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:39:02.043432    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:39:46.018879 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:39:46.018810    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:39:46.565155 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:39:46.565122    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:39:51.230408 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:39:51.230374    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5b5cc56b97-dgbdg_417b1d82-cebe-475f-9cce-c6bf2ac230fd/storage-initializer/0.log"
Jul 28 18:39:51.243145 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:39:51.243116    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5b5cc56b97-dgbdg_417b1d82-cebe-475f-9cce-c6bf2ac230fd/main/0.log"
Jul 28 18:40:06.787924 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:06.787890    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"]
Jul 28 18:40:21.811980 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:21.811906    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" containerID="cri-o://c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa" gracePeriod=45
Jul 28 18:40:22.036945 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.036876    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 28 18:40:22.475568 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.475527    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:40:22.628302 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628207    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tls-certs\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.628302 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628243    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tmp-dir\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.628302 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628277    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-model-cache\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.628547 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628389    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-dshm\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.628547 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628440    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kserve-provision-location\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.628547 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628466    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417b1d82-cebe-475f-9cce-c6bf2ac230fd-cabundle-cert\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.628547 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628510    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff2n4\" (UniqueName: \"kubernetes.io/projected/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kube-api-access-ff2n4\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.628547 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628529    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:40:22.628547 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628541    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-home\") pod \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\" (UID: \"417b1d82-cebe-475f-9cce-c6bf2ac230fd\") "
Jul 28 18:40:22.629303 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628907    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.629303 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.628909    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417b1d82-cebe-475f-9cce-c6bf2ac230fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:22.629303 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.629268    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-home" (OuterVolumeSpecName: "home") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:40:22.630755 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.630734    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-dshm" (OuterVolumeSpecName: "dshm") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:40:22.630859 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.630837    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:40:22.630921 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.630870    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kube-api-access-ff2n4" (OuterVolumeSpecName: "kube-api-access-ff2n4") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "kube-api-access-ff2n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:22.640472 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.640430    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:40:22.697154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.697110    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "417b1d82-cebe-475f-9cce-c6bf2ac230fd" (UID: "417b1d82-cebe-475f-9cce-c6bf2ac230fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:40:22.716369 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.716337    2585 generic.go:356] "Generic (PLEG): container finished" podID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerID="c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa" exitCode=0
Jul 28 18:40:22.716580 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.716394    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"
Jul 28 18:40:22.716580 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.716419    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" event={"ID":"417b1d82-cebe-475f-9cce-c6bf2ac230fd","Type":"ContainerDied","Data":"c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa"}
Jul 28 18:40:22.716580 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.716476    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg" event={"ID":"417b1d82-cebe-475f-9cce-c6bf2ac230fd","Type":"ContainerDied","Data":"1dcf772d4ced6846e8015754b6146ae5d017b0a1abccccae330707f319a26e3d"}
Jul 28 18:40:22.716580 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.716499    2585 scope.go:117] "RemoveContainer" containerID="c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa"
Jul 28 18:40:22.726871 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.726843    2585 scope.go:117] "RemoveContainer" containerID="6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d"
Jul 28 18:40:22.729927 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.729898    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2n4\" (UniqueName: \"kubernetes.io/projected/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kube-api-access-ff2n4\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.730034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.729941    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.730034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.729957    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.730034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.729976    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.730034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.729989    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.730034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.730008    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417b1d82-cebe-475f-9cce-c6bf2ac230fd-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.730034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.730021    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417b1d82-cebe-475f-9cce-c6bf2ac230fd-cabundle-cert\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.737826 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.737808    2585 scope.go:117] "RemoveContainer" containerID="c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa"
Jul 28 18:40:22.738069 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:40:22.738050    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa\": container with ID starting with c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa not found: ID does not exist" containerID="c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa"
Jul 28 18:40:22.738124 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.738078    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa"} err="failed to get container status \"c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa\": rpc error: code = NotFound desc = could not find container \"c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa\": container with ID starting with c012180f2967f2f354922f857cf8285364c0409fb4ec5a3a2b60c45d692593fa not found: ID does not exist"
Jul 28 18:40:22.738124 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.738095    2585 scope.go:117] "RemoveContainer" containerID="6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d"
Jul 28 18:40:22.738311 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:40:22.738297    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d\": container with ID starting with 6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d not found: ID does not exist" containerID="6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d"
Jul 28 18:40:22.738359 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.738314    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d"} err="failed to get container status \"6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d\": rpc error: code = NotFound desc = could not find container \"6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d\": container with ID starting with 6f8d2964bbb351d005f11918d83b1e8d4dd42dff3bb8a85d2a0cd6cdb1369e8d not found: ID does not exist"
Jul 28 18:40:22.746922 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.746892    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"]
Jul 28 18:40:22.753267 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:22.753241    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5b5cc56b97-dgbdg"]
Jul 28 18:40:23.181012 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:23.180974    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" path="/var/lib/kubelet/pods/417b1d82-cebe-475f-9cce-c6bf2ac230fd/volumes"
Jul 28 18:40:38.214101 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.214064    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"]
Jul 28 18:40:38.214505 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.214377    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main"
Jul 28 18:40:38.214505 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.214388    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main"
Jul 28 18:40:38.214505 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.214402    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="storage-initializer"
Jul 28 18:40:38.214505 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.214408    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="storage-initializer"
Jul 28 18:40:38.214505 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.214465    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="417b1d82-cebe-475f-9cce-c6bf2ac230fd" containerName="main"
Jul 28 18:40:38.218977 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.218956    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.221884 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.221855    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 28 18:40:38.222013 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.221948    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:40:38.222273 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.222252    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 28 18:40:38.223015 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.222988    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 28 18:40:38.223142 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.223014    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-2hh55\""
Jul 28 18:40:38.223142 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.223017    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 28 18:40:38.238773 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.238749    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"]
Jul 28 18:40:38.362837 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.362790    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.362837 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.362840    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.362837 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.362899    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tls-certs\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.363236 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.362944    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-model-cache\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.363236 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.362983    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.363236 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.363012    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-home\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.363236 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.363090    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrw7s\" (UniqueName: \"kubernetes.io/projected/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kube-api-access-zrw7s\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.363236 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.363152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-dshm\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464328 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-dshm\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464328 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464283    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464328 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464332    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tls-certs\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464361    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-model-cache\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464388    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-home\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464635 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464458    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrw7s\" (UniqueName: \"kubernetes.io/projected/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kube-api-access-zrw7s\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.464941 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464910    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-home\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.465034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.464983    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.465034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.465014    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.465278 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.465257    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-model-cache\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.465334 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.465309    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.466695 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.466671    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-dshm\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.466947 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.466921    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tls-certs\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.475496 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.475473    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrw7s\" (UniqueName: \"kubernetes.io/projected/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kube-api-access-zrw7s\") pod \"lora-multiple-adapters-test-kserve-65956d74c8-f29h2\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.529717 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.529680    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:38.683391 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.683364    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"]
Jul 28 18:40:38.685642 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:40:38.685607    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1a4496_1aa3_40b5_922b_f46c2b6b9a71.slice/crio-eb626e35aa948ae931c32256e08840606e02a6c86a483f89e2db14a4865d2403 WatchSource:0}: Error finding container eb626e35aa948ae931c32256e08840606e02a6c86a483f89e2db14a4865d2403: Status 404 returned error can't find the container with id eb626e35aa948ae931c32256e08840606e02a6c86a483f89e2db14a4865d2403
Jul 28 18:40:38.787194 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.787156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" event={"ID":"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71","Type":"ContainerStarted","Data":"8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82"}
Jul 28 18:40:38.787363 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:38.787220    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" event={"ID":"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71","Type":"ContainerStarted","Data":"eb626e35aa948ae931c32256e08840606e02a6c86a483f89e2db14a4865d2403"}
Jul 28 18:40:50.844785 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:50.844739    2585 generic.go:356] "Generic (PLEG): container finished" podID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerID="8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82" exitCode=0
Jul 28 18:40:50.845321 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:50.844807    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" event={"ID":"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71","Type":"ContainerDied","Data":"8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82"}
Jul 28 18:40:51.852637 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:51.852601    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" event={"ID":"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71","Type":"ContainerStarted","Data":"4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1"}
Jul 28 18:40:51.878626 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:51.878577    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podStartSLOduration=2.60313225 podStartE2EDuration="13.878543526s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.275411276s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:51.877093918 +0000 UTC m=+5037.220686195" watchObservedRunningTime="2026-07-28 18:40:51.878543526 +0000 UTC m=+5037.222135802"
Jul 28 18:40:52.530770 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:52.530739    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:58.530505 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:58.530460    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:40:58.532396 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:40:58.532367    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 28 18:41:08.531078 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:08.531034    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 28 18:41:18.531197 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:18.531137    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 28 18:41:28.530570 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:28.530512    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 28 18:41:38.530061 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:38.530002    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 28 18:41:48.530611 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:48.530568    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 28 18:41:49.566226 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:41:49.566186    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:41:50.114033 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:50.113999    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:41:55.420786 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:55.420659    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:41:55.439307 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:55.439283    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:41:58.531298 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:41:58.531235    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 28 18:42:08.540253 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:42:08.540218    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:42:08.548332 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:42:08.548306    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:43:04.144804 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:04.144765    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-65956d74c8-f29h2_dc1a4496-1aa3-40b5-922b-f46c2b6b9a71/storage-initializer/0.log"
Jul 28 18:43:04.157489 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:04.157458    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-65956d74c8-f29h2_dc1a4496-1aa3-40b5-922b-f46c2b6b9a71/main/0.log"
Jul 28 18:43:08.684812 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:08.684774    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"]
Jul 28 18:43:23.706981 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:23.706927    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main" containerID="cri-o://4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1" gracePeriod=45
Jul 28 18:43:24.350985 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.350960    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:43:24.485851 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.485748    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-dshm\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.485851 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.485817    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrw7s\" (UniqueName: \"kubernetes.io/projected/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kube-api-access-zrw7s\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.485851 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.485848    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-cabundle-cert\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.486153 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.485902    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tls-certs\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.486153 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.485949    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tmp-dir\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.486153 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.485981    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-model-cache\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.486153 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.486003    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-home\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.486153 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.486044    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kserve-provision-location\") pod \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\" (UID: \"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71\") "
Jul 28 18:43:24.486402 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.486249    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:43:24.486402 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.486374    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.486589 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.486528    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-home" (OuterVolumeSpecName: "home") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:43:24.486756 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.486736    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:43:24.488283 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.488251    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-dshm" (OuterVolumeSpecName: "dshm") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:43:24.488408 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.488372    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:43:24.488408 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.488385    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kube-api-access-zrw7s" (OuterVolumeSpecName: "kube-api-access-zrw7s") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "kube-api-access-zrw7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:43:24.499034 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.499004    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:43:24.556735 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.556695    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" (UID: "dc1a4496-1aa3-40b5-922b-f46c2b6b9a71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:43:24.569238 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.569205    2585 generic.go:356] "Generic (PLEG): container finished" podID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerID="4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1" exitCode=0
Jul 28 18:43:24.569387 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.569263    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"
Jul 28 18:43:24.569387 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.569255    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" event={"ID":"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71","Type":"ContainerDied","Data":"4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1"}
Jul 28 18:43:24.569464 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.569394    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2" event={"ID":"dc1a4496-1aa3-40b5-922b-f46c2b6b9a71","Type":"ContainerDied","Data":"eb626e35aa948ae931c32256e08840606e02a6c86a483f89e2db14a4865d2403"}
Jul 28 18:43:24.569464 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.569425    2585 scope.go:117] "RemoveContainer" containerID="4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1"
Jul 28 18:43:24.578731 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.578705    2585 scope.go:117] "RemoveContainer" containerID="8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82"
Jul 28 18:43:24.586788 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.586769    2585 scope.go:117] "RemoveContainer" containerID="4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1"
Jul 28 18:43:24.587045 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:43:24.587026    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1\": container with ID starting with 4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1 not found: ID does not exist" containerID="4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1"
Jul 28 18:43:24.587094 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587053    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1"} err="failed to get container status \"4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1\": rpc error: code = NotFound desc = could not find container \"4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1\": container with ID starting with 4f5e68d684fd02d4e26ba00883a8bf5bde36e98d3a0c8105ec21bddeae45e3e1 not found: ID does not exist"
Jul 28 18:43:24.587094 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587075    2585 scope.go:117] "RemoveContainer" containerID="8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82"
Jul 28 18:43:24.587200 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587160    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.587200 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587178    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.587200 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587191    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.587331 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587203    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.587331 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587218    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrw7s\" (UniqueName: \"kubernetes.io/projected/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-kube-api-access-zrw7s\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.587331 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587233    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-cabundle-cert\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.587331 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587244    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:43:24.587331 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:43:24.587317    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82\": container with ID starting with 8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82 not found: ID does not exist" containerID="8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82"
Jul 28 18:43:24.587488 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.587340    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82"} err="failed to get container status \"8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82\": rpc error: code = NotFound desc = could not find container \"8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82\": container with ID starting with 8bd168ac3ba39d36ba8b3eaf6013db98770c3f3169afea8cdb449aa7f5ba0e82 not found: ID does not exist"
Jul 28 18:43:24.593528 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.593507    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"]
Jul 28 18:43:24.597773 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:24.597749    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-65956d74c8-f29h2"]
Jul 28 18:43:25.182143 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:25.182095    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" path="/var/lib/kubelet/pods/dc1a4496-1aa3-40b5-922b-f46c2b6b9a71/volumes"
Jul 28 18:43:34.935320 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.935278    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"]
Jul 28 18:43:34.935790 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.935771    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="storage-initializer"
Jul 28 18:43:34.935839 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.935795    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="storage-initializer"
Jul 28 18:43:34.935839 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.935822    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main"
Jul 28 18:43:34.935839 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.935831    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main"
Jul 28 18:43:34.935935 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.935912    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1a4496-1aa3-40b5-922b-f46c2b6b9a71" containerName="main"
Jul 28 18:43:34.945937 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.945903    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:34.948911 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.948877    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 28 18:43:34.949901 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.949875    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 18:43:34.950521 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.950503    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 18:43:34.950651 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.950604    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-nnc8d\""
Jul 28 18:43:34.950651 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.950633    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 18:43:34.951209 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:34.951183    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"]
Jul 28 18:43:35.077059 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.077021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqtxp\" (UniqueName: \"kubernetes.io/projected/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kube-api-access-wqtxp\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.077322 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.077069    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-home\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.077322 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.077088    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-dshm\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.077322 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.077166    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-model-cache\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.077322 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.077208    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tls-certs\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.077322 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.077227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tmp-dir\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.077322 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.077275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kserve-provision-location\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.177793 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.177760    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kserve-provision-location\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.177975 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.177817    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtxp\" (UniqueName: \"kubernetes.io/projected/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kube-api-access-wqtxp\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.177975 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.177854    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-home\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.177975 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.177877    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-dshm\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.177975 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.177913    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-model-cache\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.177975 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.177959    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tls-certs\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.178262 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.177987    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tmp-dir\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.178262 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.178230    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kserve-provision-location\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.178597 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.178544    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-home\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.178739 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.178630    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-model-cache\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.178739 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.178683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tmp-dir\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.180325 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.180296    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-dshm\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.180892 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.180869    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tls-certs\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.189514 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.189460    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtxp\" (UniqueName: \"kubernetes.io/projected/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kube-api-access-wqtxp\") pod \"tls-verification-test-kserve-86d9c7d957-ht556\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.259597 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.259529    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:35.397512 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.397481    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"]
Jul 28 18:43:35.400097 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:43:35.400071    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1321ca_bb53_4102_8c63_cf5bfc7884cb.slice/crio-903d3f885f7b9deda673d5913779d9eab9400b1bc2d368188dace72b26c1d407 WatchSource:0}: Error finding container 903d3f885f7b9deda673d5913779d9eab9400b1bc2d368188dace72b26c1d407: Status 404 returned error can't find the container with id 903d3f885f7b9deda673d5913779d9eab9400b1bc2d368188dace72b26c1d407
Jul 28 18:43:35.630711 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.630676    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" event={"ID":"bb1321ca-bb53-4102-8c63-cf5bfc7884cb","Type":"ContainerStarted","Data":"007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6"}
Jul 28 18:43:35.630883 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:35.630727    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" event={"ID":"bb1321ca-bb53-4102-8c63-cf5bfc7884cb","Type":"ContainerStarted","Data":"903d3f885f7b9deda673d5913779d9eab9400b1bc2d368188dace72b26c1d407"}
Jul 28 18:43:43.670205 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:43.670167    2585 generic.go:356] "Generic (PLEG): container finished" podID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerID="007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6" exitCode=0
Jul 28 18:43:43.670641 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:43.670240    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" event={"ID":"bb1321ca-bb53-4102-8c63-cf5bfc7884cb","Type":"ContainerDied","Data":"007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6"}
Jul 28 18:43:44.677592 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:44.677543    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" event={"ID":"bb1321ca-bb53-4102-8c63-cf5bfc7884cb","Type":"ContainerStarted","Data":"81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6"}
Jul 28 18:43:44.705210 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:44.705135    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podStartSLOduration=2.646192643 podStartE2EDuration="10.705114595s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.058921952s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:43:44.702106739 +0000 UTC m=+5210.045699018" watchObservedRunningTime="2026-07-28 18:43:44.705114595 +0000 UTC m=+5210.048706874"
Jul 28 18:43:45.260207 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:45.260166    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:45.260385 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:45.260236    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:43:45.261926 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:45.261891    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:43:53.115518 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:43:53.115419    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:43:53.722530 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:53.722494    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:43:55.260862 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:43:55.260806    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:44:05.259985 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:44:05.259928    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:44:15.260523 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:44:15.260477    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:44:25.260785 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:44:25.260727    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:44:35.260620 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:44:35.260571    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:44:45.260051 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:44:45.260012    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:44:55.260353 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:44:55.260308    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:45:05.260198 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:05.260151    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:45:15.260319 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:15.260272    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:45:25.260178 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:25.260123    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:45:35.270181 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:35.270150    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:45:35.277939 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:35.277915    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:45:42.118064 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:42.118030    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"]
Jul 28 18:45:56.723100 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:45:56.723055    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:45:57.139926 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.139852    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" containerID="cri-o://81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6" gracePeriod=45
Jul 28 18:45:57.253714 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.253680    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:45:57.271453 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.271411    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 18:45:57.878751 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.878726    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:45:57.968018 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.967927    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tmp-dir\") pod \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") "
Jul 28 18:45:57.968018 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.967968    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-home\") pod \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") "
Jul 28 18:45:57.968018 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968014    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-model-cache\") pod \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") "
Jul 28 18:45:57.968293 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968049    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-dshm\") pod \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") "
Jul 28 18:45:57.968293 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968108    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqtxp\" (UniqueName: \"kubernetes.io/projected/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kube-api-access-wqtxp\") pod \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") "
Jul 28 18:45:57.968293 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968134    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tls-certs\") pod \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") "
Jul 28 18:45:57.968293 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968155    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kserve-provision-location\") pod \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\" (UID: \"bb1321ca-bb53-4102-8c63-cf5bfc7884cb\") "
Jul 28 18:45:57.968493 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968310    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb1321ca-bb53-4102-8c63-cf5bfc7884cb" (UID: "bb1321ca-bb53-4102-8c63-cf5bfc7884cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:57.968493 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968445    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:45:57.968890 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.968849    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-home" (OuterVolumeSpecName: "home") pod "bb1321ca-bb53-4102-8c63-cf5bfc7884cb" (UID: "bb1321ca-bb53-4102-8c63-cf5bfc7884cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:57.970602 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.970576    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-dshm" (OuterVolumeSpecName: "dshm") pod "bb1321ca-bb53-4102-8c63-cf5bfc7884cb" (UID: "bb1321ca-bb53-4102-8c63-cf5bfc7884cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:57.970852 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.970825    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb1321ca-bb53-4102-8c63-cf5bfc7884cb" (UID: "bb1321ca-bb53-4102-8c63-cf5bfc7884cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:45:57.970952 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.970879    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kube-api-access-wqtxp" (OuterVolumeSpecName: "kube-api-access-wqtxp") pod "bb1321ca-bb53-4102-8c63-cf5bfc7884cb" (UID: "bb1321ca-bb53-4102-8c63-cf5bfc7884cb"). InnerVolumeSpecName "kube-api-access-wqtxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:45:57.986138 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:57.986109    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb1321ca-bb53-4102-8c63-cf5bfc7884cb" (UID: "bb1321ca-bb53-4102-8c63-cf5bfc7884cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:58.034948 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.034900    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb1321ca-bb53-4102-8c63-cf5bfc7884cb" (UID: "bb1321ca-bb53-4102-8c63-cf5bfc7884cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:58.069723 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.069694    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqtxp\" (UniqueName: \"kubernetes.io/projected/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kube-api-access-wqtxp\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:45:58.069723 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.069722    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:45:58.069885 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.069733    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:45:58.069885 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.069742    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:45:58.069885 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.069750    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:45:58.069885 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.069758    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1321ca-bb53-4102-8c63-cf5bfc7884cb-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:45:58.259797 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.259706    2585 generic.go:356] "Generic (PLEG): container finished" podID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerID="81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6" exitCode=0
Jul 28 18:45:58.259797 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.259754    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"
Jul 28 18:45:58.259977 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.259795    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" event={"ID":"bb1321ca-bb53-4102-8c63-cf5bfc7884cb","Type":"ContainerDied","Data":"81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6"}
Jul 28 18:45:58.259977 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.259851    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556" event={"ID":"bb1321ca-bb53-4102-8c63-cf5bfc7884cb","Type":"ContainerDied","Data":"903d3f885f7b9deda673d5913779d9eab9400b1bc2d368188dace72b26c1d407"}
Jul 28 18:45:58.259977 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.259875    2585 scope.go:117] "RemoveContainer" containerID="81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6"
Jul 28 18:45:58.268934 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.268906    2585 scope.go:117] "RemoveContainer" containerID="007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6"
Jul 28 18:45:58.276745 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.276728    2585 scope.go:117] "RemoveContainer" containerID="81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6"
Jul 28 18:45:58.276975 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:45:58.276956    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6\": container with ID starting with 81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6 not found: ID does not exist" containerID="81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6"
Jul 28 18:45:58.277025 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.276982    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6"} err="failed to get container status \"81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6\": rpc error: code = NotFound desc = could not find container \"81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6\": container with ID starting with 81c757128afbe1ceade8397652c42e52fc43ba0b40ab97a4730190f71b040ad6 not found: ID does not exist"
Jul 28 18:45:58.277025 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.276997    2585 scope.go:117] "RemoveContainer" containerID="007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6"
Jul 28 18:45:58.277192 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:45:58.277176    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6\": container with ID starting with 007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6 not found: ID does not exist" containerID="007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6"
Jul 28 18:45:58.277233 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.277196    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6"} err="failed to get container status \"007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6\": rpc error: code = NotFound desc = could not find container \"007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6\": container with ID starting with 007ac550ce3a519412c23fca881d8b7772bd422b2ef32a8b179351f2a4e241e6 not found: ID does not exist"
Jul 28 18:45:58.294192 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.294170    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"]
Jul 28 18:45:58.309367 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:58.309347    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d9c7d957-ht556"]
Jul 28 18:45:59.180470 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:45:59.180433    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" path="/var/lib/kubelet/pods/bb1321ca-bb53-4102-8c63-cf5bfc7884cb/volumes"
Jul 28 18:46:18.320018 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.319984    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"]
Jul 28 18:46:18.320406 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.320326    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main"
Jul 28 18:46:18.320406 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.320338    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main"
Jul 28 18:46:18.320406 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.320353    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="storage-initializer"
Jul 28 18:46:18.320406 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.320360    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="storage-initializer"
Jul 28 18:46:18.320543 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.320412    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1321ca-bb53-4102-8c63-cf5bfc7884cb" containerName="main"
Jul 28 18:46:18.323788 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.323767    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.326495 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.326464    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-brxdv\""
Jul 28 18:46:18.326652 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.326604    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 18:46:18.326652 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.326618    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 18:46:18.326652 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.326627    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 28 18:46:18.326818 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.326690    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 18:46:18.335432 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.335406    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"]
Jul 28 18:46:18.440334 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.440293    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-model-cache\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.440502 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.440407    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tmp-dir\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.440502 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.440443    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g25fx\" (UniqueName: \"kubernetes.io/projected/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kube-api-access-g25fx\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.440502 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.440486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-home\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.440671 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.440509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tls-certs\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.440671 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.440631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-dshm\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.440750 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.440669    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kserve-provision-location\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541277 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541237    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tmp-dir\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541455 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541285    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25fx\" (UniqueName: \"kubernetes.io/projected/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kube-api-access-g25fx\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541455 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541319    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-home\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541455 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tls-certs\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541455 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541357    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-dshm\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541718 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541489    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kserve-provision-location\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541718 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-model-cache\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541718 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541688    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tmp-dir\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541844 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541727    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-home\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541887 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541860    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kserve-provision-location\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.541926 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.541897    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-model-cache\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.544181 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.544157    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-dshm\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.544395 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.544380    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tls-certs\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.551321 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.551301    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25fx\" (UniqueName: \"kubernetes.io/projected/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kube-api-access-g25fx\") pod \"prestop-hook-test-kserve-648d46f47f-c865f\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.635473 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.635386    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:18.769797 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:18.769766    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"]
Jul 28 18:46:18.773375 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:46:18.773347    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7d4a85_aab8_43d7_843a_92d307c46a9d.slice/crio-24a959032b09e3ecac9e1a07a33d8b7eb7f0f94d4f3bca769db7bb3666800dc8 WatchSource:0}: Error finding container 24a959032b09e3ecac9e1a07a33d8b7eb7f0f94d4f3bca769db7bb3666800dc8: Status 404 returned error can't find the container with id 24a959032b09e3ecac9e1a07a33d8b7eb7f0f94d4f3bca769db7bb3666800dc8
Jul 28 18:46:19.349836 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:19.349791    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" event={"ID":"ee7d4a85-aab8-43d7-843a-92d307c46a9d","Type":"ContainerStarted","Data":"4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76"}
Jul 28 18:46:19.350315 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:19.349866    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" event={"ID":"ee7d4a85-aab8-43d7-843a-92d307c46a9d","Type":"ContainerStarted","Data":"24a959032b09e3ecac9e1a07a33d8b7eb7f0f94d4f3bca769db7bb3666800dc8"}
Jul 28 18:46:48.477878 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:48.477841    2585 generic.go:356] "Generic (PLEG): container finished" podID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerID="4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76" exitCode=0
Jul 28 18:46:48.478431 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:48.477905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" event={"ID":"ee7d4a85-aab8-43d7-843a-92d307c46a9d","Type":"ContainerDied","Data":"4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76"}
Jul 28 18:46:49.488308 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:49.488272    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" event={"ID":"ee7d4a85-aab8-43d7-843a-92d307c46a9d","Type":"ContainerStarted","Data":"11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0"}
Jul 28 18:46:49.513431 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:49.513366    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podStartSLOduration=2.461714069 podStartE2EDuration="31.513345021s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.051630952s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:46:49.509609309 +0000 UTC m=+5394.853201587" watchObservedRunningTime="2026-07-28 18:46:49.513345021 +0000 UTC m=+5394.856937300"
Jul 28 18:46:49.635685 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:49.635645    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:55.451884 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:55.451758    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:46:55.469071 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:55.469043    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:46:58.635831 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:58.635782    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:46:58.637197 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:46:58.637165    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:47:08.636584 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:47:08.636496    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:47:18.636865 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:47:18.636807    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:47:28.636208 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:47:28.636153    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:47:38.636563 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:47:38.636512    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:47:48.636245 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:47:48.636194    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:47:58.636603 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:47:58.636538    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:48:00.254671 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:48:00.254618    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:48:00.811100 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:00.811063    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:48:08.636525 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:08.636471    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:48:18.636116 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:18.636002    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:48:28.636213 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:28.636168    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:48:38.635848 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:38.635800    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 28 18:48:48.646176 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:48.646145    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:48:48.654300 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:48.654268    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:48:52.945383 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:48:52.945338    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"]
Jul 28 18:49:07.967239 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:07.967201    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" containerID="cri-o://11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0" gracePeriod=45
Jul 28 18:49:08.717963 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.717940    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:49:08.845960 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.845928    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-model-cache\") pod \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") "
Jul 28 18:49:08.846154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.845986    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-home\") pod \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") "
Jul 28 18:49:08.846154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846032    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-dshm\") pod \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") "
Jul 28 18:49:08.846154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846061    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tmp-dir\") pod \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") "
Jul 28 18:49:08.846154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846090    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g25fx\" (UniqueName: \"kubernetes.io/projected/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kube-api-access-g25fx\") pod \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") "
Jul 28 18:49:08.846154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846114    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kserve-provision-location\") pod \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") "
Jul 28 18:49:08.846154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846146    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tls-certs\") pod \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\" (UID: \"ee7d4a85-aab8-43d7-843a-92d307c46a9d\") "
Jul 28 18:49:08.846490 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846277    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee7d4a85-aab8-43d7-843a-92d307c46a9d" (UID: "ee7d4a85-aab8-43d7-843a-92d307c46a9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:08.846490 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846395    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:49:08.846798 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.846764    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-home" (OuterVolumeSpecName: "home") pod "ee7d4a85-aab8-43d7-843a-92d307c46a9d" (UID: "ee7d4a85-aab8-43d7-843a-92d307c46a9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:08.848417 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.848386    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-dshm" (OuterVolumeSpecName: "dshm") pod "ee7d4a85-aab8-43d7-843a-92d307c46a9d" (UID: "ee7d4a85-aab8-43d7-843a-92d307c46a9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:08.848775 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.848754    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee7d4a85-aab8-43d7-843a-92d307c46a9d" (UID: "ee7d4a85-aab8-43d7-843a-92d307c46a9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:49:08.848865 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.848784    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kube-api-access-g25fx" (OuterVolumeSpecName: "kube-api-access-g25fx") pod "ee7d4a85-aab8-43d7-843a-92d307c46a9d" (UID: "ee7d4a85-aab8-43d7-843a-92d307c46a9d"). InnerVolumeSpecName "kube-api-access-g25fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:49:08.858805 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.858773    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee7d4a85-aab8-43d7-843a-92d307c46a9d" (UID: "ee7d4a85-aab8-43d7-843a-92d307c46a9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:08.916171 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.916103    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee7d4a85-aab8-43d7-843a-92d307c46a9d" (UID: "ee7d4a85-aab8-43d7-843a-92d307c46a9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:08.947241 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.947212    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:49:08.947241 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.947239    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:49:08.947424 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.947252    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:49:08.947424 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.947267    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g25fx\" (UniqueName: \"kubernetes.io/projected/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kube-api-access-g25fx\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:49:08.947424 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.947280    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee7d4a85-aab8-43d7-843a-92d307c46a9d-kserve-provision-location\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:49:08.947424 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:08.947292    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d4a85-aab8-43d7-843a-92d307c46a9d-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:49:09.085161 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.085123    2585 generic.go:356] "Generic (PLEG): container finished" podID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerID="11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0" exitCode=0
Jul 28 18:49:09.085639 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.085192    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"
Jul 28 18:49:09.085639 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.085210    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" event={"ID":"ee7d4a85-aab8-43d7-843a-92d307c46a9d","Type":"ContainerDied","Data":"11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0"}
Jul 28 18:49:09.085639 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.085295    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" event={"ID":"ee7d4a85-aab8-43d7-843a-92d307c46a9d","Type":"ContainerDied","Data":"24a959032b09e3ecac9e1a07a33d8b7eb7f0f94d4f3bca769db7bb3666800dc8"}
Jul 28 18:49:09.085639 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.085329    2585 scope.go:117] "RemoveContainer" containerID="11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0"
Jul 28 18:49:09.094824 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.094807    2585 scope.go:117] "RemoveContainer" containerID="4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76"
Jul 28 18:49:09.102604 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.102587    2585 scope.go:117] "RemoveContainer" containerID="11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0"
Jul 28 18:49:09.102862 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:49:09.102845    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0\": container with ID starting with 11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0 not found: ID does not exist" containerID="11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0"
Jul 28 18:49:09.102904 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.102869    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0"} err="failed to get container status \"11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0\": rpc error: code = NotFound desc = could not find container \"11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0\": container with ID starting with 11648736d64ec48b58fdabdd40808ab7d62cdc9135ce63717bb03418c854fff0 not found: ID does not exist"
Jul 28 18:49:09.102904 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.102885    2585 scope.go:117] "RemoveContainer" containerID="4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76"
Jul 28 18:49:09.103154 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:49:09.103138    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76\": container with ID starting with 4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76 not found: ID does not exist" containerID="4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76"
Jul 28 18:49:09.103207 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.103156    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76"} err="failed to get container status \"4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76\": rpc error: code = NotFound desc = could not find container \"4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76\": container with ID starting with 4b46a070a19ba25c811745fade7c1392b59e82aee533390f5a8f10ce7c158b76 not found: ID does not exist"
Jul 28 18:49:09.110169 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.110142    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"]
Jul 28 18:49:09.114046 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.114023    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f"]
Jul 28 18:49:09.179721 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.179694    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" path="/var/lib/kubelet/pods/ee7d4a85-aab8-43d7-843a-92d307c46a9d/volumes"
Jul 28 18:49:09.647711 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:49:09.647672    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-648d46f47f-c865f" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Jul 28 18:50:03.811973 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:50:03.811919    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:50:04.304205 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:04.304174    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:50:26.922399 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.922365    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"]
Jul 28 18:50:26.923077 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.922844    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main"
Jul 28 18:50:26.923077 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.922864    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main"
Jul 28 18:50:26.923077 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.922900    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="storage-initializer"
Jul 28 18:50:26.923077 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.922909    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="storage-initializer"
Jul 28 18:50:26.923077 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.922994    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee7d4a85-aab8-43d7-843a-92d307c46a9d" containerName="main"
Jul 28 18:50:26.926317 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.926296    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:26.930316 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.930290    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-52lvb\""
Jul 28 18:50:26.930464 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.930337    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 18:50:26.930464 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.930290    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 18:50:26.930636 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.930519    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 18:50:26.930711 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.930696    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-xwwh4\""
Jul 28 18:50:26.935802 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:26.935782    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"]
Jul 28 18:50:27.002110 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.002077    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:27.002271 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.002140    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk4c\" (UniqueName: \"kubernetes.io/projected/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-kube-api-access-6rk4c\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:27.102809 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.102772    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk4c\" (UniqueName: \"kubernetes.io/projected/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-kube-api-access-6rk4c\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:27.102980 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.102871    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:27.105361 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.105340    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:27.140128 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.140104    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk4c\" (UniqueName: \"kubernetes.io/projected/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-kube-api-access-6rk4c\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:27.237802 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.237722    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:27.364965 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.364933    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"]
Jul 28 18:50:27.366910 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:50:27.366883    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ddc386_21c8_468b_a1d7_5dfbf2dfc99a.slice/crio-627f8e48aee33cc2d787aef0822cb9f9cf718067263218026cc3124daf8c5897 WatchSource:0}: Error finding container 627f8e48aee33cc2d787aef0822cb9f9cf718067263218026cc3124daf8c5897: Status 404 returned error can't find the container with id 627f8e48aee33cc2d787aef0822cb9f9cf718067263218026cc3124daf8c5897
Jul 28 18:50:27.395752 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:27.395720    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5" event={"ID":"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a","Type":"ContainerStarted","Data":"627f8e48aee33cc2d787aef0822cb9f9cf718067263218026cc3124daf8c5897"}
Jul 28 18:50:28.401311 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:28.401274    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5" event={"ID":"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a","Type":"ContainerStarted","Data":"42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e"}
Jul 28 18:50:28.401757 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:28.401340    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:50:28.419799 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:28.419745    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5" podStartSLOduration=2.4197307 podStartE2EDuration="2.4197307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:50:28.417892633 +0000 UTC m=+5613.761484993" watchObservedRunningTime="2026-07-28 18:50:28.4197307 +0000 UTC m=+5613.763322997"
Jul 28 18:50:59.409991 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:50:59.409960    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:51:14.304999 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.304968    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"]
Jul 28 18:51:14.308377 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.308357    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.316517 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.316491    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"]
Jul 28 18:51:14.404366 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.404330    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.404532 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.404379    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshhh\" (UniqueName: \"kubernetes.io/projected/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-kube-api-access-jshhh\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.404532 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.404464    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.404532 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.404511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.404675 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.404614    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.404675 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.404639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505261 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505231    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505439 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505275    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505439 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505343    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505585 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505476    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505585 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505707 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshhh\" (UniqueName: \"kubernetes.io/projected/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-kube-api-access-jshhh\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505707 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505647    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505808 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505752    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.505878 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.505856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.507756 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.507731    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.507883 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.507835    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.513712 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.513692    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshhh\" (UniqueName: \"kubernetes.io/projected/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-kube-api-access-jshhh\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-j8kl4\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.618374 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.618284    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:14.750943 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:14.750915    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"]
Jul 28 18:51:14.753194 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:51:14.753159    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d42cac_0033_4e0d_baed_f9b07adf5f2c.slice/crio-c118d4534cc3082979915cb6ec65810181c85d31e52bdcddca15397eb8301029 WatchSource:0}: Error finding container c118d4534cc3082979915cb6ec65810181c85d31e52bdcddca15397eb8301029: Status 404 returned error can't find the container with id c118d4534cc3082979915cb6ec65810181c85d31e52bdcddca15397eb8301029
Jul 28 18:51:15.598660 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:15.598621    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" event={"ID":"b7d42cac-0033-4e0d-baed-f9b07adf5f2c","Type":"ContainerStarted","Data":"d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3"}
Jul 28 18:51:15.599059 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:15.598703    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" event={"ID":"b7d42cac-0033-4e0d-baed-f9b07adf5f2c","Type":"ContainerStarted","Data":"c118d4534cc3082979915cb6ec65810181c85d31e52bdcddca15397eb8301029"}
Jul 28 18:51:15.617392 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:15.617344    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" podStartSLOduration=1.617331097 podStartE2EDuration="1.617331097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:51:15.615009546 +0000 UTC m=+5660.958601823" watchObservedRunningTime="2026-07-28 18:51:15.617331097 +0000 UTC m=+5660.960923373"
Jul 28 18:51:15.619168 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:15.619144    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:23.821086 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:23.821044    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"]
Jul 28 18:51:23.832364 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:23.831006    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"]
Jul 28 18:51:23.834817 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:51:23.834743    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:51:23.834817 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:51:23.834817 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 18:51:23.834817 ip-10-0-135-20 kubenswrapper[2585]:  > containerID="d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3" cmd=["/bin/sleep","15"]
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:23.834999    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" message="" err=<
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]:  >
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:51:23.835027    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" podUID="b7d42cac-0033-4e0d-baed-f9b07adf5f2c" containerName="main" containerID="cri-o://d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3"
Jul 28 18:51:23.835067 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:23.835063    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" podUID="b7d42cac-0033-4e0d-baed-f9b07adf5f2c" containerName="main" containerID="cri-o://d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3" gracePeriod=60
Jul 28 18:51:24.076314 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.076246    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:24.194275 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194242    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tmp-dir\") pod \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") "
Jul 28 18:51:24.194473 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194286    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-model-cache\") pod \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") "
Jul 28 18:51:24.194473 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194326    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-dshm\") pod \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") "
Jul 28 18:51:24.194473 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194346    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tls-certs\") pod \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") "
Jul 28 18:51:24.194473 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194373    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-home\") pod \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") "
Jul 28 18:51:24.194473 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194422    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jshhh\" (UniqueName: \"kubernetes.io/projected/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-kube-api-access-jshhh\") pod \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\" (UID: \"b7d42cac-0033-4e0d-baed-f9b07adf5f2c\") "
Jul 28 18:51:24.194755 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194596    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7d42cac-0033-4e0d-baed-f9b07adf5f2c" (UID: "b7d42cac-0033-4e0d-baed-f9b07adf5f2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:24.194755 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194709    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-home" (OuterVolumeSpecName: "home") pod "b7d42cac-0033-4e0d-baed-f9b07adf5f2c" (UID: "b7d42cac-0033-4e0d-baed-f9b07adf5f2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:24.194755 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194721    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7d42cac-0033-4e0d-baed-f9b07adf5f2c" (UID: "b7d42cac-0033-4e0d-baed-f9b07adf5f2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:24.194755 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.194727    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tmp-dir\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:24.196741 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.196720    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-dshm" (OuterVolumeSpecName: "dshm") pod "b7d42cac-0033-4e0d-baed-f9b07adf5f2c" (UID: "b7d42cac-0033-4e0d-baed-f9b07adf5f2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:24.196741 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.196725    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-kube-api-access-jshhh" (OuterVolumeSpecName: "kube-api-access-jshhh") pod "b7d42cac-0033-4e0d-baed-f9b07adf5f2c" (UID: "b7d42cac-0033-4e0d-baed-f9b07adf5f2c"). InnerVolumeSpecName "kube-api-access-jshhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:51:24.196835 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.196775    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7d42cac-0033-4e0d-baed-f9b07adf5f2c" (UID: "b7d42cac-0033-4e0d-baed-f9b07adf5f2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:51:24.295572 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.295507    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-model-cache\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:24.295572 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.295542    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-dshm\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:24.295763 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.295581    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:24.295763 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.295605    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-home\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:24.295763 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.295617    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshhh\" (UniqueName: \"kubernetes.io/projected/b7d42cac-0033-4e0d-baed-f9b07adf5f2c-kube-api-access-jshhh\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:24.641753 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.641716    2585 generic.go:356] "Generic (PLEG): container finished" podID="b7d42cac-0033-4e0d-baed-f9b07adf5f2c" containerID="d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3" exitCode=0
Jul 28 18:51:24.641965 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.641783    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"
Jul 28 18:51:24.641965 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.641791    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" event={"ID":"b7d42cac-0033-4e0d-baed-f9b07adf5f2c","Type":"ContainerDied","Data":"d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3"}
Jul 28 18:51:24.641965 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.641908    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4" event={"ID":"b7d42cac-0033-4e0d-baed-f9b07adf5f2c","Type":"ContainerDied","Data":"c118d4534cc3082979915cb6ec65810181c85d31e52bdcddca15397eb8301029"}
Jul 28 18:51:24.641965 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.641930    2585 scope.go:117] "RemoveContainer" containerID="d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3"
Jul 28 18:51:24.651995 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.651972    2585 scope.go:117] "RemoveContainer" containerID="d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3"
Jul 28 18:51:24.652269 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:51:24.652246    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3\": container with ID starting with d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3 not found: ID does not exist" containerID="d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3"
Jul 28 18:51:24.652375 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.652277    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3"} err="failed to get container status \"d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3\": rpc error: code = NotFound desc = could not find container \"d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3\": container with ID starting with d3c86889a0fbdba75f3fe4f4e124f8cd47a188dc5bc7d14a1b9c46121c88fee3 not found: ID does not exist"
Jul 28 18:51:24.667009 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.666983    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"]
Jul 28 18:51:24.670693 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:24.670664    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-j8kl4"]
Jul 28 18:51:25.180060 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:25.180030    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d42cac-0033-4e0d-baed-f9b07adf5f2c" path="/var/lib/kubelet/pods/b7d42cac-0033-4e0d-baed-f9b07adf5f2c/volumes"
Jul 28 18:51:29.409350 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:29.409310    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5" podUID="c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:51:38.853974 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:38.853894    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5" podUID="c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" containerName="main" containerID="cri-o://42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e" gracePeriod=45
Jul 28 18:51:39.108416 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.108361    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:51:39.220009 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.219976    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rk4c\" (UniqueName: \"kubernetes.io/projected/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-kube-api-access-6rk4c\") pod \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") "
Jul 28 18:51:39.220176 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.220015    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-tls-certs\") pod \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\" (UID: \"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a\") "
Jul 28 18:51:39.222190 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.222158    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-kube-api-access-6rk4c" (OuterVolumeSpecName: "kube-api-access-6rk4c") pod "c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" (UID: "c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a"). InnerVolumeSpecName "kube-api-access-6rk4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:51:39.222190 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.222166    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" (UID: "c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:51:39.321027 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.320997    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rk4c\" (UniqueName: \"kubernetes.io/projected/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-kube-api-access-6rk4c\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:39.321187 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.321112    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a-tls-certs\") on node \"ip-10-0-135-20.ec2.internal\" DevicePath \"\""
Jul 28 18:51:39.717640 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.717602    2585 generic.go:356] "Generic (PLEG): container finished" podID="c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" containerID="42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e" exitCode=0
Jul 28 18:51:39.717640 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.717642    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5" event={"ID":"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a","Type":"ContainerDied","Data":"42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e"}
Jul 28 18:51:39.717868 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.717678    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5" event={"ID":"c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a","Type":"ContainerDied","Data":"627f8e48aee33cc2d787aef0822cb9f9cf718067263218026cc3124daf8c5897"}
Jul 28 18:51:39.717868 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.717700    2585 scope.go:117] "RemoveContainer" containerID="42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e"
Jul 28 18:51:39.717868 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.717645    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"
Jul 28 18:51:39.726769 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.726751    2585 scope.go:117] "RemoveContainer" containerID="42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e"
Jul 28 18:51:39.726995 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:51:39.726978    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e\": container with ID starting with 42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e not found: ID does not exist" containerID="42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e"
Jul 28 18:51:39.727043 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.727003    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e"} err="failed to get container status \"42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e\": rpc error: code = NotFound desc = could not find container \"42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e\": container with ID starting with 42cfb3f704fa44829d18d475d3f952c8c44ef3988534346e88d1c5997680c91e not found: ID does not exist"
Jul 28 18:51:39.741338 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.741312    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"]
Jul 28 18:51:39.745195 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:39.745172    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-bnbm5"]
Jul 28 18:51:41.180047 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:41.180012    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" path="/var/lib/kubelet/pods/c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a/volumes"
Jul 28 18:51:55.476520 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:55.476415    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:51:55.494777 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:51:55.494755    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:52:07.305316 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:52:07.305269    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:52:07.835824 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:52:07.835784    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:54:10.836357 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:54:10.836310    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:54:11.305778 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:54:11.305743    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:55:23.203830 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:23.203768    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/0.log"
Jul 28 18:55:23.261106 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:23.261054    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/1.log"
Jul 28 18:55:23.283754 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:23.283717    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dc97f78c-hsl5f_3cc057ee-b81c-4977-ba8a-b8a2f478729b/router/0.log"
Jul 28 18:55:24.605589 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:24.605542    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/0.log"
Jul 28 18:55:24.663107 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:24.663067    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/1.log"
Jul 28 18:55:24.698310 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:24.698274    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dc97f78c-hsl5f_3cc057ee-b81c-4977-ba8a-b8a2f478729b/router/0.log"
Jul 28 18:55:26.189519 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:26.189482    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/0.log"
Jul 28 18:55:26.244724 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:26.244694    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/1.log"
Jul 28 18:55:26.268280 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:26.268256    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dc97f78c-hsl5f_3cc057ee-b81c-4977-ba8a-b8a2f478729b/router/0.log"
Jul 28 18:55:27.607327 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:27.607296    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/0.log"
Jul 28 18:55:27.665071 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:27.665036    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/1.log"
Jul 28 18:55:27.683970 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:27.683947    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dc97f78c-hsl5f_3cc057ee-b81c-4977-ba8a-b8a2f478729b/router/0.log"
Jul 28 18:55:28.585407 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:28.585370    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:28.628964 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:28.628928    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:29.758340 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:29.758302    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:29.816568 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:29.816533    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:30.946680 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:30.946643    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:30.995879 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:30.995851    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:32.095822 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:32.095766    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:32.139833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:32.139801    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:33.237769 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:33.237737    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:33.297679 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:33.297652    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:34.390251 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:34.390213    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:34.439532 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:34.439505    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:35.543868 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:35.543833    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:35.592001 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:35.591977    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:36.642846 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:36.642815    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:36.783530 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:36.783503    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:37.882140 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:37.882103    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:37.934091 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:37.934063    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:39.056122 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:39.056077    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:39.101806 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:39.101777    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:40.263570 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:40.263521    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:40.307123 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:40.307095    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:41.405978 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:41.405942    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:41.456991 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:41.456963    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:42.567728 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:42.567691    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:42.617027 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:42.617001    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:43.727278 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:43.727239    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:43.777628 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:43.777603    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:44.880100 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:44.880063    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:44.932994 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:44.932971    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:46.044692 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:46.044657    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:46.091687 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:46.091659    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:47.196468 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:47.196386    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:47.241301 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:47.241278    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:48.327657 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:48.327623    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:48.371915 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:48.371887    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:49.451533 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:49.451495    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:49.496668 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:49.496640    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:50.633655 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:50.633617    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:50.739910 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:50.739886    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:51.840654 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:51.840616    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:51.890874 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:51.890845    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:52.999053 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:52.999019    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:53.045165 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:53.045140    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:54.132838 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:54.132802    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:54.187779 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:54.187751    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:55.301887 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:55.301854    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:55.351331 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:55.351299    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:56.445356 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:56.445323    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:56.491594 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:56.491568    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:57.578260 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:57.578223    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:57.625983 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:57.625954    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:58.718074 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:58.718035    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:58.763319 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:58.763291    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:55:59.845920 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:59.845887    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:55:59.896713 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:55:59.896686    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:01.058192 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:01.058153    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:01.112267 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:01.112235    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:02.215400 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:02.215361    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:02.265116 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:02.265088    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:03.356964 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:03.356928    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:03.406073 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:03.406045    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:04.522833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:04.522800    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:04.570426 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:04.570392    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:05.708698 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:05.708663    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:05.753269 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:05.753240    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:06.862378 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:06.862334    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:06.910090 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:06.910065    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:07.996801 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:07.996765    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:08.039427 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:08.039400    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:09.117291 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:09.117256    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:09.168770 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:09.168744    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:10.255106 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:10.255068    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:10.299817 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:10.299788    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:14.306741 ip-10-0-135-20 kubenswrapper[2585]: E0728 18:56:14.306694    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs" podUID="50febc0e-e107-4f9e-9767-670777188d7c"
Jul 28 18:56:14.812321 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:14.812286    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs"
Jul 28 18:56:15.489189 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:15.489152    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-27fcp_8f4bfba2-733c-453a-a03d-02d4994797d4/global-pull-secret-syncer/0.log"
Jul 28 18:56:15.707730 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:15.707700    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fz847_0a5be7c1-fe99-43a9-acd3-912ea1a5c61e/konnectivity-agent/0.log"
Jul 28 18:56:15.800808 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:15.800778    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-20.ec2.internal_0dd2e9a077ed553dd3893a28c3e87410/haproxy/0.log"
Jul 28 18:56:20.013702 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:20.013666    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-85tfv_417fb782-62e8-4d22-bb74-0105b1d425c6/authorino/0.log"
Jul 28 18:56:20.113933 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:20.113901    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrjg8_4449761e-d0aa-46cd-97ff-c2d1639e37a3/kuadrant-console-plugin/0.log"
Jul 28 18:56:22.372988 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:22.372958    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8mhj_a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3/node-exporter/0.log"
Jul 28 18:56:22.392627 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:22.392600    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8mhj_a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3/kube-rbac-proxy/0.log"
Jul 28 18:56:22.413420 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:22.413400    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8mhj_a3b3c2e6-55fc-4c66-ab16-c5a46ae2bbd3/init-textfile/0.log"
Jul 28 18:56:22.534489 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:22.534461    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4fth6_99103575-d3da-4012-b08a-ec2204751a9f/kube-rbac-proxy-main/0.log"
Jul 28 18:56:22.556869 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:22.556840    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4fth6_99103575-d3da-4012-b08a-ec2204751a9f/kube-rbac-proxy-self/0.log"
Jul 28 18:56:22.576147 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:22.576120    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4fth6_99103575-d3da-4012-b08a-ec2204751a9f/openshift-state-metrics/0.log"
Jul 28 18:56:26.103832 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.103788    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"]
Jul 28 18:56:26.104342 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.104101    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d42cac-0033-4e0d-baed-f9b07adf5f2c" containerName="main"
Jul 28 18:56:26.104342 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.104112    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d42cac-0033-4e0d-baed-f9b07adf5f2c" containerName="main"
Jul 28 18:56:26.104342 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.104125    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" containerName="main"
Jul 28 18:56:26.104342 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.104131    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" containerName="main"
Jul 28 18:56:26.104342 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.104182    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d42cac-0033-4e0d-baed-f9b07adf5f2c" containerName="main"
Jul 28 18:56:26.104342 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.104193    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ddc386-21c8-468b-a1d7-5dfbf2dfc99a" containerName="main"
Jul 28 18:56:26.106963 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.106947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.109754 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.109731    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ct488\"/\"openshift-service-ca.crt\""
Jul 28 18:56:26.109899 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.109803    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ct488\"/\"kube-root-ca.crt\""
Jul 28 18:56:26.109899 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.109884    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ct488\"/\"default-dockercfg-gl4lk\""
Jul 28 18:56:26.116835 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.116815    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"]
Jul 28 18:56:26.215734 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.215702    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-podres\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.215892 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.215740    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-proc\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.215892 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.215762    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-lib-modules\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.215892 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.215829    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-sys\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.215892 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.215870    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjv8\" (UniqueName: \"kubernetes.io/projected/e001eead-09f4-462e-8e8f-cd52dd0a3d72-kube-api-access-mcjv8\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316647 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316618    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-podres\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316654    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-proc\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316676    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-lib-modules\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316704    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-sys\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316726    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjv8\" (UniqueName: \"kubernetes.io/projected/e001eead-09f4-462e-8e8f-cd52dd0a3d72-kube-api-access-mcjv8\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316764    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-proc\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316784    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-podres\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.316833 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316815    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-lib-modules\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.317100 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.316842    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e001eead-09f4-462e-8e8f-cd52dd0a3d72-sys\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.325478 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.325457    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjv8\" (UniqueName: \"kubernetes.io/projected/e001eead-09f4-462e-8e8f-cd52dd0a3d72-kube-api-access-mcjv8\") pod \"perf-node-gather-daemonset-rndv9\" (UID: \"e001eead-09f4-462e-8e8f-cd52dd0a3d72\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.416946 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.416870    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.547572 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.547453    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"]
Jul 28 18:56:26.549885 ip-10-0-135-20 kubenswrapper[2585]: W0728 18:56:26.549858    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode001eead_09f4_462e_8e8f_cd52dd0a3d72.slice/crio-3b7f705672f0ea40eff00e8a2b49ba3eff21a91845b30745ce5eea019430b8c6 WatchSource:0}: Error finding container 3b7f705672f0ea40eff00e8a2b49ba3eff21a91845b30745ce5eea019430b8c6: Status 404 returned error can't find the container with id 3b7f705672f0ea40eff00e8a2b49ba3eff21a91845b30745ce5eea019430b8c6
Jul 28 18:56:26.864250 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.864211    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9" event={"ID":"e001eead-09f4-462e-8e8f-cd52dd0a3d72","Type":"ContainerStarted","Data":"06e732482f39365a5ed516a4dd5a2741ebda95e0a1364655b1f4cc09de0eef7c"}
Jul 28 18:56:26.864415 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.864268    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9" event={"ID":"e001eead-09f4-462e-8e8f-cd52dd0a3d72","Type":"ContainerStarted","Data":"3b7f705672f0ea40eff00e8a2b49ba3eff21a91845b30745ce5eea019430b8c6"}
Jul 28 18:56:26.864415 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.864296    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:26.883319 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:26.883271    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9" podStartSLOduration=0.883256892 podStartE2EDuration="883.256892ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:56:26.881878346 +0000 UTC m=+5972.225470623" watchObservedRunningTime="2026-07-28 18:56:26.883256892 +0000 UTC m=+5972.226849168"
Jul 28 18:56:27.494124 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:27.494087    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cqcfv_def621fc-443a-4f17-8c74-bbe3605a4c89/dns/0.log"
Jul 28 18:56:27.520486 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:27.520460    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cqcfv_def621fc-443a-4f17-8c74-bbe3605a4c89/kube-rbac-proxy/0.log"
Jul 28 18:56:27.650956 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:27.650925    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gwcq_07aa6d61-639a-4114-9f14-21ef6bfe55fd/dns-node-resolver/0.log"
Jul 28 18:56:28.328523 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:28.328492    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7nqzl_c3e7698f-f10c-4566-9052-349b0610b252/node-ca/0.log"
Jul 28 18:56:29.891126 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:29.891092    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/0.log"
Jul 28 18:56:29.947016 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:29.946978    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/1.log"
Jul 28 18:56:29.972166 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:29.972121    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dc97f78c-hsl5f_3cc057ee-b81c-4977-ba8a-b8a2f478729b/router/0.log"
Jul 28 18:56:30.524248 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:30.524217    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sd8qz_e537ee6c-b73a-46e2-8e85-9d4dce87ea88/serve-healthcheck-canary/0.log"
Jul 28 18:56:31.012092 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:31.011992    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bttvz_7bb9c484-525b-430e-bee3-69208d19d269/kube-rbac-proxy/0.log"
Jul 28 18:56:31.034448 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:31.034424    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bttvz_7bb9c484-525b-430e-bee3-69208d19d269/exporter/0.log"
Jul 28 18:56:31.055595 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:31.055567    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bttvz_7bb9c484-525b-430e-bee3-69208d19d269/extractor/0.log"
Jul 28 18:56:32.877804 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:32.877777    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-rndv9"
Jul 28 18:56:34.067253 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:34.067222    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8994644bb-4xntv_f461e503-ea75-4f23-ae41-a85adacc2826/manager/0.log"
Jul 28 18:56:34.796812 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:34.796777    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85d8b8ddfc-svxs4_6d705ac4-a201-43fa-abba-168b7af1d0ac/manager/0.log"
Jul 28 18:56:35.536719 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:35.536689    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jlqc6_8b31043f-82fa-4ce2-8bf1-703ce0eef666/s3-init/0.log"
Jul 28 18:56:35.555266 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:35.555239    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jlqc6_8b31043f-82fa-4ce2-8bf1-703ce0eef666/download-hf-model/0.log"
Jul 28 18:56:36.157147 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:36.157094    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/0.log"
Jul 28 18:56:36.216248 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:36.216207    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/1.log"
Jul 28 18:56:36.228389 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:36.228359    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dc97f78c-hsl5f_3cc057ee-b81c-4977-ba8a-b8a2f478729b/router/0.log"
Jul 28 18:56:37.813744 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:37.813711    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/0.log"
Jul 28 18:56:37.877290 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:37.877262    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l_c4c71c04-b83e-44b3-b3ce-4a06dfec9307/istio-proxy/1.log"
Jul 28 18:56:37.896161 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:37.896134    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dc97f78c-hsl5f_3cc057ee-b81c-4977-ba8a-b8a2f478729b/router/0.log"
Jul 28 18:56:40.891604 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:40.891570    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xrcn8_4eb017f2-945a-4a40-a843-d1ae387db67e/migrator/0.log"
Jul 28 18:56:40.911571 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:40.911516    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xrcn8_4eb017f2-945a-4a40-a843-d1ae387db67e/graceful-termination/0.log"
Jul 28 18:56:41.274166 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:41.274128    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf_76fd56c1-4eba-4445-a7b5-a75f304f150b/kube-storage-version-migrator-operator/1.log"
Jul 28 18:56:41.275898 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:41.275865    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf_76fd56c1-4eba-4445-a7b5-a75f304f150b/kube-storage-version-migrator-operator/0.log"
Jul 28 18:56:42.345914 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.345886    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xzdj_91aee170-ad83-440e-8544-ccc56a8f216a/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:56:42.372081 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.372049    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xzdj_91aee170-ad83-440e-8544-ccc56a8f216a/egress-router-binary-copy/0.log"
Jul 28 18:56:42.393835 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.393800    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xzdj_91aee170-ad83-440e-8544-ccc56a8f216a/cni-plugins/0.log"
Jul 28 18:56:42.416526 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.416498    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xzdj_91aee170-ad83-440e-8544-ccc56a8f216a/bond-cni-plugin/0.log"
Jul 28 18:56:42.438135 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.438103    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xzdj_91aee170-ad83-440e-8544-ccc56a8f216a/routeoverride-cni/0.log"
Jul 28 18:56:42.460417 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.460386    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xzdj_91aee170-ad83-440e-8544-ccc56a8f216a/whereabouts-cni-bincopy/0.log"
Jul 28 18:56:42.484353 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.484322    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xzdj_91aee170-ad83-440e-8544-ccc56a8f216a/whereabouts-cni/0.log"
Jul 28 18:56:42.981714 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:42.981681    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f6vs6_1b95f9c4-e82e-4530-bfb1-740d8c4e8c22/kube-multus/0.log"
Jul 28 18:56:43.123631 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:43.123591    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ng94v_0a3ff851-6c38-46db-b020-57ae09a57aa3/network-metrics-daemon/0.log"
Jul 28 18:56:43.142661 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:43.142632    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ng94v_0a3ff851-6c38-46db-b020-57ae09a57aa3/kube-rbac-proxy/0.log"
Jul 28 18:56:44.229717 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.229690    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-controller/0.log"
Jul 28 18:56:44.246747 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.246713    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/1.log"
Jul 28 18:56:44.246747 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.246727    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovn-acl-logging/0.log"
Jul 28 18:56:44.269154 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.269130    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/kube-rbac-proxy-node/0.log"
Jul 28 18:56:44.290448 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.290424    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:56:44.309765 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.309738    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/northd/0.log"
Jul 28 18:56:44.332130 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.332109    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/nbdb/0.log"
Jul 28 18:56:44.353945 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.353923    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/sbdb/0.log"
Jul 28 18:56:44.445613 ip-10-0-135-20 kubenswrapper[2585]: I0728 18:56:44.445579    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxrll_3a35d96c-75c0-4979-9470-4a4fc570ed7d/ovnkube-controller/0.log"