Sep 18 19:36:08.154063 ip-10-0-133-25 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:36:08.154077 ip-10-0-133-25 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:36:08.154086 ip-10-0-133-25 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:36:08.154403 ip-10-0-133-25 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:36:18.359250 ip-10-0-133-25 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:36:18.359264 ip-10-0-133-25 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7eee12bcc0f54ed1a235da360e2f47dd --
Sep 18 19:38:24.279728 ip-10-0-133-25 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:38:27.734765 ip-10-0-133-25 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:27.734765 ip-10-0-133-25 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:38:27.734765 ip-10-0-133-25 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:27.734765 ip-10-0-133-25 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:38:27.734765 ip-10-0-133-25 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:27.735584 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.734853    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:38:27.739394 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739374    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:27.739394 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739393    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:27.739394 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739398    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:27.739394 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739402    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739407    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739411    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739416    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739420    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739424    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739428    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739431    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739435    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739439    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739443    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739448    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739452    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739456    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739460    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739464    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739467    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739471    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739475    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739479    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:27.739643 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739482    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739486    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739490    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739494    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739497    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739501    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739506    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739510    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739514    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739518    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739522    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739526    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739530    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739534    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739539    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739543    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739548    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739552    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739556    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739560    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:27.740470 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739564    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739570    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739574    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739578    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739583    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739587    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739592    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739596    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739600    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739604    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739609    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739612    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739617    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739621    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739627    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739631    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739635    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739640    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739644    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739648    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:27.741069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739652    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739659    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739665    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739670    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739675    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739680    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739686    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739690    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739694    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739699    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739704    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739708    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739712    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739717    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739722    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739727    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739733    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739738    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739742    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:27.741591 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739746    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739750    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739759    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739765    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739769    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739774    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739779    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739783    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739788    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739792    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739796    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739800    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739805    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739811    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739816    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739821    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:27.742068 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.739826    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:27.742493 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742478    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:27.742493 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742493    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742498    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742503    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742507    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742510    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742513    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742516    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742519    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742522    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742524    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742527    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742530    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742532    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742534    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742537    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742541    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742545    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742547    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:27.742550 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742550    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742552    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742555    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742558    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742561    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742564    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742568    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742571    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742574    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742577    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742580    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742583    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742585    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742589    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742591    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742594    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742597    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742599    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742602    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742604    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:27.743028 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742607    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742609    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742611    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742614    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742616    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742618    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742621    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742623    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742625    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742628    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742630    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742633    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742636    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742638    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742642    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742645    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742647    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742649    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742652    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742655    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:27.743543 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742657    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742660    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742662    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742665    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742668    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742670    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742672    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742675    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742677    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742679    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742682    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742685    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742687    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742689    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742691    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742694    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742696    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742699    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742702    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742704    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:27.744037 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742706    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742709    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742711    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742714    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742717    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742720    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742723    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742726    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742728    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742731    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742733    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742736    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742738    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742741    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742743    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742746    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742748    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742750    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742753    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.742755    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:27.744553 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743127    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743132    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743135    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743138    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743150    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743154    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743158    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743160    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743163    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743166    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743168    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743171    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743174    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743177    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743179    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743182    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743185    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743187    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743190    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743193    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:27.745040 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743195    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743198    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743200    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743203    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743205    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743207    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743210    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743213    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743217    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743219    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743221    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743224    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743226    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743229    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743231    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743234    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743236    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743238    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743240    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:27.745560 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743243    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743246    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743248    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743250    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743253    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743256    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743260    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743262    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743265    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743267    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743270    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743273    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743275    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743277    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743280    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743282    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743285    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743288    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743290    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743292    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:27.746027 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743295    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743298    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743300    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743302    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743305    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743307    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743310    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743312    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743314    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743317    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743319    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743321    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743324    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743326    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743328    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743333    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743336    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743339    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743345    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743348    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:27.746532 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743370    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743375    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743379    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743386    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743389    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743391    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743394    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743397    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743400    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743402    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743405    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743407    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743411    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743415    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743418    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743421    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743424    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743426    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743429    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.743431    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:27.747016 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743505    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743539    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743544    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743549    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743553    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743556    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743560    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743566    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743570    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743572    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743577    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743580    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743583    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743586    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743588    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743591    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743595    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743598    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743602    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743605    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743608    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743611    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743614    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743623    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:38:27.747520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743626    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743629    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743632    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743635    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743638    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743641    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743644    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743646    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743650    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743653    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743656    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743659    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743662    2583 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743665    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743669    2583 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743673    2583 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743676    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743679    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743683    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743686    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743689    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743692    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743695    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743698    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743701    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:38:27.748120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743704    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743707    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743710    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743713    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743716    2583 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743719    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743722    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743725    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743728    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743731    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743734    2583 flags.go:64] FLAG: --help="false"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743737    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-133-25.ec2.internal"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743740    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743743    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743746    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743749    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743752    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743755    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743758    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743760    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743763    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743766    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743770    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743773    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:38:27.748747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743776    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743780    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743783    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743786    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743788    2583 flags.go:64] FLAG: --lock-file=""
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743791    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743794    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743797    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743802    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743805    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743808    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743811    2583 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743813    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743817    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743819    2583 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743822    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743826    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743829    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743833    2583 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743836    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743839    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743842    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743845    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743848    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743851    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:38:27.749319 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743854    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743861    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743864    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743867    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743869    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743874    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743879    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743882    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743885    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743890    2583 flags.go:64] FLAG: --port="10250"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743893    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743895    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04634b351a650e37f"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743899    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743901    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743904    2583 flags.go:64] FLAG: --register-node="true"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743907    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743910    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743914    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743917    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743919    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743923    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743926    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743929    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743931    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743934    2583 flags.go:64] FLAG: --runonce="false"
Sep 18 19:38:27.749947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743937    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743940    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743942    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743945    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743948    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743950    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743953    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743956    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743959    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743961    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743964    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743966    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743971    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743973    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743976    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743981    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743986    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743988    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743993    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743996    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.743999    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.744001    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.744004    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.744007    2583 flags.go:64] FLAG: --v="2"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.744014    2583 flags.go:64] FLAG: --version="false"
Sep 18 19:38:27.750620 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.744019    2583 flags.go:64] FLAG: --vmodule=""
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.744023    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.744026    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744153    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744157    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744160    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744163    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744166    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744169    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744171    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744174    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744176    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744179    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744182    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744184    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744187    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744190    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744192    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744194    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744199    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:27.751252 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744201    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744204    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744206    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744212    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744215    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744218    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744221    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744223    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744226    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744229    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744232    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744234    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744237    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744240    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744242    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744245    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744248    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744250    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744252    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744255    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:27.751752 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744257    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744260    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744262    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744265    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744267    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744270    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744272    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744274    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744277    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744279    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744282    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744286    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744288    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744291    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744293    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744297    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744300    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744302    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744305    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744307    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:27.752241 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744310    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744313    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744315    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744318    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744320    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744323    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744326    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744329    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744331    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744334    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744336    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744339    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744341    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744344    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744346    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744349    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744366    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744369    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744372    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:27.752753 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744376    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744379    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744382    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744385    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744389    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744392    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744395    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744397    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744402    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744404    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744407    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744409    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744412    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744414    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744417    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744419    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744422    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744424    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744427    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744429    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:27.753212 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744432    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744435    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.744439    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.745145    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.752140    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.752157    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.752177    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.752185    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752241    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752246    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752251    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752255    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752259    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752262    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:27.753707 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752265    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752268    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752270    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752273    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752276    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752278    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752281    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752283    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752286    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752289    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752295    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752297    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752300    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752303    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752305    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752308    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752311    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752313    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752316    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:27.754062 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752319    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752321    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752324    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752327    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752330    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752332    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752335    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752337    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752340    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752342    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752344    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752347    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752349    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752366    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752369    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752371    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752374    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752376    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752378    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752381    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:27.754572 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752385    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752389    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752392    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752394    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752397    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752399    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752402    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752404    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752407    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752409    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752412    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752414    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752417    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752419    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752422    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752424    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752427    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752430    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752433    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752435    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:27.755073 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752437    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752440    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752442    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752445    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752448    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752450    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752453    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752455    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752459    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752462    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752465    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752468    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752471    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752474    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752477    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752479    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752482    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752484    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752487    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752489    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:27.755585 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752492    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752494    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752496    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752499    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752501    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752504    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752506    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752509    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752511    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752514    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752517    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752519    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752522    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.752527    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752639    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:27.756072 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752643    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752646    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752649    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752652    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752655    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752659    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752662    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752664    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752667    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752670    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752672    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752675    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752678    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752680    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752682    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752685    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752687    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752690    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752693    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752695    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:27.756481 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752698    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752700    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752703    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752705    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752708    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752710    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752713    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752716    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752719    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752721    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752724    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752726    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752729    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752731    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752733    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752736    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752738    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752741    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752744    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752746    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:27.756971 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752749    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752751    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752753    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752756    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752759    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752761    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752764    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752767    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752769    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752771    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752774    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752776    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752779    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752781    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752784    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752786    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752788    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752791    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752793    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:27.757475 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752796    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752799    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752802    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752804    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752807    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752810    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752813    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752816    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752819    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752821    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752824    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752826    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752828    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752831    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752833    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752836    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752838    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752840    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752843    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752845    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:27.757969 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752848    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752851    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752855    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752858    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752860    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752863    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752866    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752869    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752871    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752874    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752876    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752879    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752882    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752884    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752888    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752890    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752893    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752895    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:27.758588 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:27.752898    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:27.759008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.752903    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:27.759008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.753009    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:38:27.759008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.754889    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:38:27.759008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.755738    2583 server.go:1019] "Starting client certificate rotation"
Sep 18 19:38:27.759008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.755831    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:27.759008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.756533    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:27.759188 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.759105    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:27.760221 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.760196    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:27.766418 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.766256    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:38:27.768107 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.768094    2583 log.go:25] "Validated CRI v1 image API"
Sep 18 19:38:27.770222 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.770201    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:38:27.772788 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.772769    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a46dd714-c018-444a-9818-1166a287174f:/dev/nvme0n1p3 d4b478ec-c065-4239-b6a1-72188c439197:/dev/nvme0n1p4]
Sep 18 19:38:27.772855 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.772789    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:38:27.777080 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.776972    2583 manager.go:216] Machine: {Timestamp:2026-09-18 19:38:27.77648205 +0000 UTC m=+2.030688646 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3168081 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28479298d0d5d2bbaaae0a7b87d4b4 SystemUUID:ec284792-98d0-d5d2-bbaa-ae0a7b87d4b4 BootID:7eee12bc-c0f5-4ed1-a235-da360e2f47dd Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:da:4d:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:da:4d:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:71:eb:4b:aa:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:38:27.777080 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.777074    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:38:27.777176 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.777146    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:38:27.778626 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.778605    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:38:27.778762 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.778629    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-25.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:38:27.779967 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.779957    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:38:27.780000 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.779971    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:38:27.780000 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.779978    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:27.780000 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.779988    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:27.780083 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.780052    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:38:27.783801 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.783789    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:27.783906 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.783896    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:38:27.783935 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.783928    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:38:27.783962 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.783938    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:38:27.783962 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.783955    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:38:27.784014 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.783966    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:38:27.784014 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.783974    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:38:27.784347 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.784333    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:27.784453 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.784369    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:27.786386 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.786346    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:38:27.786458 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.786415    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:27.788490 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.788476    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:38:27.788543 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.788507    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:38:27.789448 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789437    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:38:27.789482 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789453    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:38:27.789482 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789459    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:38:27.789482 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789464    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:38:27.789482 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789474    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789482    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789502    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789509    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789517    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789522    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789531    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789536    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789563    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:38:27.789590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.789568    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:38:27.793325 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.793314    2583 server.go:1268] "Started kubelet"
Sep 18 19:38:27.793471 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.793456    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-25.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:38:27.793530 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.793515    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:38:27.793621 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.793608    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:38:27.793675 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.793660    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:38:27.793992 ip-10-0-133-25 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:38:27.794666 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.794604    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:38:27.795556 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.795517    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:38:27.795664 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.795581    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:38:27.797842 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.797827    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:38:27.800232 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.800216    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:38:27.800306 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.800222    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:27.825640 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.825613    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:27.825850 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.825520    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:38:27.825947 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.825936    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:38:27.826572 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.826557    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:38:27.826694 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.826678    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:38:27.826770 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.826562    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5xf6"
Sep 18 19:38:27.826958 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.826944    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5xf6"
Sep 18 19:38:27.828119 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.826719    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:38:27.828187 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.827673    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:38:27.828187 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.827762    2583 factory.go:55] Registering systemd factory
Sep 18 19:38:27.828187 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.828164    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:38:27.828380 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.828329    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:38:27.828507 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.828487    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:38:27.828643 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.828623    2583 factory.go:153] Registering CRI-O factory
Sep 18 19:38:27.828643 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.828638    2583 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:38:27.828783 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.828724    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:38:27.828783 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.828744    2583 factory.go:103] Registering Raw factory
Sep 18 19:38:27.828783 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.828760    2583 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:38:27.829208 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.829189    2583 manager.go:349] Starting recovery of all containers
Sep 18 19:38:27.838621 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.838600    2583 manager.go:354] Recovery completed
Sep 18 19:38:27.842482 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.842468    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:27.844484 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.844469    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:27.844556 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.844503    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:27.844556 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.844518    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:27.845031 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.845018    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:38:27.845031 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.845031    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:38:27.845149 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.845048    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:27.845149 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.845032    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:27.845741 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.845724    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:27.845810 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.845754    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:27.845810 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.845766    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:27.846137 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.846126    2583 policy_none.go:49] "None policy: Start"
Sep 18 19:38:27.846178 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.846143    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:38:27.846178 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.846154    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:38:27.847047 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.847036    2583 policy_none.go:47] "Start"
Sep 18 19:38:27.886936 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.886921    2583 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.886968    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.886981    2583 server.go:87] "Starting device plugin registration server"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.887245    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.887264    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.887345    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.887440    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.887451    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.887949    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:38:27.896241 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.887997    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:27.959116 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.959072    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:38:27.960412 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.960394    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:38:27.960493 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.960436    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:38:27.960493 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.960460    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:38:27.960558 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.960500    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:38:27.962747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.962731    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:27.988230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.988186    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:27.989500 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.989485    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:27.989570 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.989512    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:27.989570 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.989521    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:27.989570 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.989554    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-25.ec2.internal"
Sep 18 19:38:27.998828 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:27.998815    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-25.ec2.internal"
Sep 18 19:38:27.998874 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:27.998835    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-25.ec2.internal\": node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.048216 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.048194    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.060770 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.060749    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal"]
Sep 18 19:38:28.060849 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.060834    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:28.061709 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.061694    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:28.061795 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.061726    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:28.061795 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.061740    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:28.063350 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.063335    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:28.063465 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.063448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.063465 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.063479    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:28.065080 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.065066    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:28.065142 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.065093    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:28.065142 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.065105    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:28.065142 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.065111    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:28.065142 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.065133    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:28.065270 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.065146    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:28.066640 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.066626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.066703 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.066649    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:28.067377 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.067347    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:28.067430 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.067395    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:28.067430 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.067409    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:28.083887 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.083866    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-25.ec2.internal\" not found" node="ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.087987 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.087959    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-25.ec2.internal\" not found" node="ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.148967 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.148944    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.229547 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.229508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62a84dcb18dfedc7e4fcb971d5de5a7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal\" (UID: \"62a84dcb18dfedc7e4fcb971d5de5a7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.229547 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.229542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a84dcb18dfedc7e4fcb971d5de5a7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal\" (UID: \"62a84dcb18dfedc7e4fcb971d5de5a7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.229734 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.229568    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3fba69533f2373986d0e5941a0f2828d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-25.ec2.internal\" (UID: \"3fba69533f2373986d0e5941a0f2828d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.249605 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.249545    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.330335 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.330310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62a84dcb18dfedc7e4fcb971d5de5a7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal\" (UID: \"62a84dcb18dfedc7e4fcb971d5de5a7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.330437 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.330312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62a84dcb18dfedc7e4fcb971d5de5a7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal\" (UID: \"62a84dcb18dfedc7e4fcb971d5de5a7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.330437 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.330376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a84dcb18dfedc7e4fcb971d5de5a7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal\" (UID: \"62a84dcb18dfedc7e4fcb971d5de5a7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.330437 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.330397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3fba69533f2373986d0e5941a0f2828d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-25.ec2.internal\" (UID: \"3fba69533f2373986d0e5941a0f2828d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.330557 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.330425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3fba69533f2373986d0e5941a0f2828d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-25.ec2.internal\" (UID: \"3fba69533f2373986d0e5941a0f2828d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.330557 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.330458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a84dcb18dfedc7e4fcb971d5de5a7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal\" (UID: \"62a84dcb18dfedc7e4fcb971d5de5a7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.350402 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.350383    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.387567 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.387539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.390958 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.390932    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:28.450794 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.450772    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.551417 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.551389    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.652042 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.652012    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.752730 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.752707    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.755870 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.755853    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:38:28.756010 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.755991    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:28.812662 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.812604    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:28.832675 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.832639    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:33:27 +0000 UTC" deadline="2028-03-22 01:34:17.328244205 +0000 UTC"
Sep 18 19:38:28.832675 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.832671    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13205h55m48.495575907s"
Sep 18 19:38:28.853820 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.853800    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:28.869126 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:28.869110    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:28.954431 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:28.954398    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:29.032400 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:29.032347    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a84dcb18dfedc7e4fcb971d5de5a7e.slice/crio-cd7d6f04c08a6877470b0a85cfcbcc30613225de2226dd251bc1a0851f88a9de WatchSource:0}: Error finding container cd7d6f04c08a6877470b0a85cfcbcc30613225de2226dd251bc1a0851f88a9de: Status 404 returned error can't find the container with id cd7d6f04c08a6877470b0a85cfcbcc30613225de2226dd251bc1a0851f88a9de
Sep 18 19:38:29.032726 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:29.032708    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fba69533f2373986d0e5941a0f2828d.slice/crio-b3f79c25b2e602e13680ca69036f61211932067bc4477186dfa19db05344d5ae WatchSource:0}: Error finding container b3f79c25b2e602e13680ca69036f61211932067bc4477186dfa19db05344d5ae: Status 404 returned error can't find the container with id b3f79c25b2e602e13680ca69036f61211932067bc4477186dfa19db05344d5ae
Sep 18 19:38:29.038263 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.038248    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:38:29.054966 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.054945    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-25.ec2.internal\" not found"
Sep 18 19:38:29.084782 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.084731    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:29.126597 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.126571    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:29.136659 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.136642    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:29.137520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.137508    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal"
Sep 18 19:38:29.146340 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.146328    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:29.784643 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.784593    2583 apiserver.go:52] "Watching apiserver"
Sep 18 19:38:29.791437 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.791412    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:38:29.791850 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.791818    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls","openshift-multus/multus-s5xwm","openshift-multus/network-metrics-daemon-q65cx","openshift-ovn-kubernetes/ovnkube-node-qzl5s","kube-system/konnectivity-agent-n9s2s","openshift-cluster-node-tuning-operator/tuned-bb2kg","openshift-dns/node-resolver-spvxq","openshift-image-registry/node-ca-fx5vm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal","openshift-multus/multus-additional-cni-plugins-w4rdh","openshift-network-diagnostics/network-check-target-52cq7","openshift-network-operator/iptables-alerter-5x6hj","kube-system/global-pull-secret-syncer-pkqrg"]
Sep 18 19:38:29.794760 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.794101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.797310 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.796804    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:29.797310 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.796843    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:29.797310 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.796998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.797310 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.797015    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvjv9\""
Sep 18 19:38:29.799642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.799205    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:29.799642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.799285    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:29.799642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.799323    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:38:29.799642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.799496    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cfz6\""
Sep 18 19:38:29.800619 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.800106    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.800619 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.800445    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:29.802166 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.802147    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:29.802415 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.802251    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:29.804344 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.804317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4bh2j\""
Sep 18 19:38:29.804464 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.804408    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:38:29.804525 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.804487    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:38:29.804582 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.804321    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:38:29.804759 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.804740    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:38:29.806328 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.806304    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.809107 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.808250    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:29.809107 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.808651    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hmlj7\""
Sep 18 19:38:29.809107 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.808909    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:38:29.809307 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.809281    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:38:29.809307 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.809286    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:38:29.809476 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.809459    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:38:29.809583 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.809565    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:38:29.809642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.809619    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:38:29.811234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.811216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j4xl2\""
Sep 18 19:38:29.811781 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.811435    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:38:29.811781 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.811626    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:38:29.812312 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.812273    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:29.813121 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.813103    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.813502 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.813482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.813599 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.813565    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:29.815577 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.815545    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:38:29.815577 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.815575    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:38:29.815749 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.815731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.815811 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.815786    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-27rzb\""
Sep 18 19:38:29.815872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.815808    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:38:29.818140 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.818113    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:38:29.818140 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.818137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.818283 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.818149    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:38:29.818283 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.818207    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:38:29.818593 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.818576    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zh2j5\""
Sep 18 19:38:29.820511 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.820489    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:38:29.820603 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.820519    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlf2\""
Sep 18 19:38:29.820669 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.820489    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:38:29.820669 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.820643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.822877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.822857    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:38:29.822974 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.822916    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jnq68\""
Sep 18 19:38:29.823033 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.822968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:29.823085 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.823035    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:29.823085 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.823072    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:38:29.829492 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.829474    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:38:29.836526 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.836505    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhldd"
Sep 18 19:38:29.840195 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.840287 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-kubelet\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.840287 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3f524e7-7958-47df-8762-0839d84b63fd-host\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.840287 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3f524e7-7958-47df-8762-0839d84b63fd-serviceca\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.840287 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-kubelet\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-sys-fs\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-modprobe-d\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840371    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysconfig\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-etc-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840453    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-registration-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:29.840519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wljr\" (UniqueName: \"kubernetes.io/projected/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-kube-api-access-7wljr\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4c8r\" (UniqueName: \"kubernetes.io/projected/a3f524e7-7958-47df-8762-0839d84b63fd-kube-api-access-l4c8r\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840568    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-node-log\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-lib-modules\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-cni-netd\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovn-node-metrics-cert\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovnkube-script-lib\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3356ecda-7f08-4f66-a08e-2cae194cc7a0-agent-certs\") pod \"konnectivity-agent-n9s2s\" (UID: \"3356ecda-7f08-4f66-a08e-2cae194cc7a0\") " pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-tmp-dir\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-system-cni-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-cnibin\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-cni-binary-copy\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-multus-certs\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.840894 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-system-cni-dir\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-systemd-units\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.840989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-slash\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysctl-d\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-cni-bin\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4fd05598-f32e-42b9-82a7-cb695c62d50c-dbus\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-var-lib-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-ovn\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-env-overrides\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841178    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-kubernetes\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-tmp\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-log-socket\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-sys\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8lj\" (UniqueName: \"kubernetes.io/projected/8e640a31-2d0a-4a6f-ae71-f3df042960f8-kube-api-access-tq8lj\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgjb5\" (UniqueName: \"kubernetes.io/projected/116348dd-64ad-47d4-a5a8-b41540fd7a67-kube-api-access-sgjb5\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3356ecda-7f08-4f66-a08e-2cae194cc7a0-konnectivity-ca\") pod \"konnectivity-agent-n9s2s\" (UID: \"3356ecda-7f08-4f66-a08e-2cae194cc7a0\") " pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:29.841610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-socket-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfd6\" (UniqueName: \"kubernetes.io/projected/5ef6b572-90dc-4447-bc19-93f0367e9926-kube-api-access-trfd6\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-k8s-cni-cncf-io\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-etc-kubernetes\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-cnibin\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-run\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-host\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4fd05598-f32e-42b9-82a7-cb695c62d50c-kubelet-config\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-cni-bin\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-device-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bcps\" (UniqueName: \"kubernetes.io/projected/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-kube-api-access-8bcps\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e640a31-2d0a-4a6f-ae71-f3df042960f8-host-slash\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-daemon-config\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-run-netns\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.842483 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovnkube-config\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-tuned\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.841993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-netns\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842063    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-systemd\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-socket-dir-parent\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842107    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-hostroot\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-systemd\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-hosts-file\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-var-lib-kubelet\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e640a31-2d0a-4a6f-ae71-f3df042960f8-iptables-alerter-script\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-cni-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-cni-multus\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsdg\" (UniqueName: \"kubernetes.io/projected/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-kube-api-access-2vsdg\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr84m\" (UniqueName: \"kubernetes.io/projected/0a3770c7-042c-40d6-954d-90f726d05bc2-kube-api-access-nr84m\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.843212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.844051 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysctl-conf\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.844051 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrcs\" (UniqueName: \"kubernetes.io/projected/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-kube-api-access-gvrcs\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.844051 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-conf-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.844051 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.842550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.844987 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.844969    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhldd"
Sep 18 19:38:29.942873 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-systemd\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.942873 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-socket-dir-parent\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.942873 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-hostroot\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.942873 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-systemd\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-hosts-file\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-var-lib-kubelet\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e640a31-2d0a-4a6f-ae71-f3df042960f8-iptables-alerter-script\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-cni-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.942984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-cni-multus\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsdg\" (UniqueName: \"kubernetes.io/projected/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-kube-api-access-2vsdg\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr84m\" (UniqueName: \"kubernetes.io/projected/0a3770c7-042c-40d6-954d-90f726d05bc2-kube-api-access-nr84m\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysctl-conf\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrcs\" (UniqueName: \"kubernetes.io/projected/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-kube-api-access-gvrcs\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-conf-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-kubelet\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3f524e7-7958-47df-8762-0839d84b63fd-host\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3f524e7-7958-47df-8762-0839d84b63fd-serviceca\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.943234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-cni-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-cni-multus\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-socket-dir-parent\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-systemd\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943303    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-hostroot\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-systemd\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-hosts-file\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-var-lib-kubelet\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.943833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysctl-conf\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e640a31-2d0a-4a6f-ae71-f3df042960f8-iptables-alerter-script\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-kubelet\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-sys-fs\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-modprobe-d\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysconfig\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-etc-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-registration-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.944070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wljr\" (UniqueName: \"kubernetes.io/projected/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-kube-api-access-7wljr\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-conf-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4c8r\" (UniqueName: \"kubernetes.io/projected/a3f524e7-7958-47df-8762-0839d84b63fd-kube-api-access-l4c8r\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-node-log\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-lib-modules\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-cni-netd\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovn-node-metrics-cert\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovnkube-script-lib\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3356ecda-7f08-4f66-a08e-2cae194cc7a0-agent-certs\") pod \"konnectivity-agent-n9s2s\" (UID: \"3356ecda-7f08-4f66-a08e-2cae194cc7a0\") " pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-tmp-dir\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944244    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3f524e7-7958-47df-8762-0839d84b63fd-host\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-system-cni-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-system-cni-dir\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-cnibin\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-sys-fs\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.944376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-cni-binary-copy\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-multus-certs\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-modprobe-d\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-system-cni-dir\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-systemd-units\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-system-cni-dir\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-slash\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysctl-d\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-cni-bin\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4fd05598-f32e-42b9-82a7-cb695c62d50c-dbus\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-var-lib-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-ovn\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944628    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-env-overrides\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-kubernetes\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-tmp\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-log-socket\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.945230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-sys\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8lj\" (UniqueName: \"kubernetes.io/projected/8e640a31-2d0a-4a6f-ae71-f3df042960f8-kube-api-access-tq8lj\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgjb5\" (UniqueName: \"kubernetes.io/projected/116348dd-64ad-47d4-a5a8-b41540fd7a67-kube-api-access-sgjb5\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3356ecda-7f08-4f66-a08e-2cae194cc7a0-konnectivity-ca\") pod \"konnectivity-agent-n9s2s\" (UID: \"3356ecda-7f08-4f66-a08e-2cae194cc7a0\") " pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944873    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-socket-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfd6\" (UniqueName: \"kubernetes.io/projected/5ef6b572-90dc-4447-bc19-93f0367e9926-kube-api-access-trfd6\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-kubelet\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-k8s-cni-cncf-io\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-etc-kubernetes\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-cnibin\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.944993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-cnibin\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3f524e7-7958-47df-8762-0839d84b63fd-serviceca\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-run\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-host\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.946108 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-var-lib-cni-bin\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4fd05598-f32e-42b9-82a7-cb695c62d50c-kubelet-config\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysconfig\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-cni-bin\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-device-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-device-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4fd05598-f32e-42b9-82a7-cb695c62d50c-dbus\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bcps\" (UniqueName: \"kubernetes.io/projected/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-kube-api-access-8bcps\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e640a31-2d0a-4a6f-ae71-f3df042960f8-host-slash\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-daemon-config\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-run-netns\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovnkube-config\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-cni-binary-copy\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-tuned\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.946978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-netns\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-multus-certs\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-cni-netd\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-etc-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.945607    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-registration-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a3770c7-042c-40d6-954d-90f726d05bc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.945693    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret podName:4fd05598-f32e-42b9-82a7-cb695c62d50c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.445647711 +0000 UTC m=+4.699854311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret") pod "global-pull-secret-syncer-pkqrg" (UID: "4fd05598-f32e-42b9-82a7-cb695c62d50c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.945696    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-var-lib-openvswitch\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.945753    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs podName:2f45aa15-f96d-4fc2-8c7b-4e96873d6147 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.445738271 +0000 UTC m=+4.699944854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs") pod "network-metrics-daemon-q65cx" (UID: "2f45aa15-f96d-4fc2-8c7b-4e96873d6147") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-run-ovn\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.945970    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-env-overrides\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-kubernetes\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.948153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-node-log\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-lib-modules\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-log-socket\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.946885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-sys\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-host\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4fd05598-f32e-42b9-82a7-cb695c62d50c-kubelet-config\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ef6b572-90dc-4447-bc19-93f0367e9926-socket-dir\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-systemd-units\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.943233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-kubelet\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a3770c7-042c-40d6-954d-90f726d05bc2-cnibin\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-tmp-dir\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947628    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-slash\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947653    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-k8s-cni-cncf-io\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947688    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-etc-kubernetes\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-run\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.949916 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947617    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e640a31-2d0a-4a6f-ae71-f3df042960f8-host-slash\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-sysctl-d\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-run-netns\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116348dd-64ad-47d4-a5a8-b41540fd7a67-host-cni-bin\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.947750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-host-run-netns\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.948084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovnkube-script-lib\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.948188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovnkube-config\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.948238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-multus-daemon-config\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.948450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3356ecda-7f08-4f66-a08e-2cae194cc7a0-konnectivity-ca\") pod \"konnectivity-agent-n9s2s\" (UID: \"3356ecda-7f08-4f66-a08e-2cae194cc7a0\") " pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.949865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116348dd-64ad-47d4-a5a8-b41540fd7a67-ovn-node-metrics-cert\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.951473 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.950110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-tmp\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.952843 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.952325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr84m\" (UniqueName: \"kubernetes.io/projected/0a3770c7-042c-40d6-954d-90f726d05bc2-kube-api-access-nr84m\") pod \"multus-additional-cni-plugins-w4rdh\" (UID: \"0a3770c7-042c-40d6-954d-90f726d05bc2\") " pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:29.953544 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.953498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrcs\" (UniqueName: \"kubernetes.io/projected/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-kube-api-access-gvrcs\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.953677 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.953652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd281f38-1f69-48e1-ac2a-f8b3cdd3b100-etc-tuned\") pod \"tuned-bb2kg\" (UID: \"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:29.954248 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.954094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3356ecda-7f08-4f66-a08e-2cae194cc7a0-agent-certs\") pod \"konnectivity-agent-n9s2s\" (UID: \"3356ecda-7f08-4f66-a08e-2cae194cc7a0\") " pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:29.954334 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.954271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsdg\" (UniqueName: \"kubernetes.io/projected/d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73-kube-api-access-2vsdg\") pod \"multus-s5xwm\" (UID: \"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73\") " pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:29.955198 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.955157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4c8r\" (UniqueName: \"kubernetes.io/projected/a3f524e7-7958-47df-8762-0839d84b63fd-kube-api-access-l4c8r\") pod \"node-ca-fx5vm\" (UID: \"a3f524e7-7958-47df-8762-0839d84b63fd\") " pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:29.956035 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.956013    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:29.956123 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.956044    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:29.956123 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.956059    2583 projected.go:196] Error preparing data for projected volume kube-api-access-48cql for pod openshift-network-diagnostics/network-check-target-52cq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:29.956123 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:29.956115    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql podName:8492f48b-11e8-4c35-88ea-d9eaf0f6de53 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.456096323 +0000 UTC m=+4.710302922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-48cql" (UniqueName: "kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql") pod "network-check-target-52cq7" (UID: "8492f48b-11e8-4c35-88ea-d9eaf0f6de53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:29.957063 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.956927    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgjb5\" (UniqueName: \"kubernetes.io/projected/116348dd-64ad-47d4-a5a8-b41540fd7a67-kube-api-access-sgjb5\") pod \"ovnkube-node-qzl5s\" (UID: \"116348dd-64ad-47d4-a5a8-b41540fd7a67\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:29.957063 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.957025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wljr\" (UniqueName: \"kubernetes.io/projected/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-kube-api-access-7wljr\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:29.957765 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.957745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8lj\" (UniqueName: \"kubernetes.io/projected/8e640a31-2d0a-4a6f-ae71-f3df042960f8-kube-api-access-tq8lj\") pod \"iptables-alerter-5x6hj\" (UID: \"8e640a31-2d0a-4a6f-ae71-f3df042960f8\") " pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:29.959470 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.959443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bcps\" (UniqueName: \"kubernetes.io/projected/077e0f30-ad3e-44a5-bf8d-4cf88903e1b0-kube-api-access-8bcps\") pod \"node-resolver-spvxq\" (UID: \"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0\") " pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:29.959737 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.959712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfd6\" (UniqueName: \"kubernetes.io/projected/5ef6b572-90dc-4447-bc19-93f0367e9926-kube-api-access-trfd6\") pod \"aws-ebs-csi-driver-node-s8qls\" (UID: \"5ef6b572-90dc-4447-bc19-93f0367e9926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:29.966861 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.966781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal" event={"ID":"62a84dcb18dfedc7e4fcb971d5de5a7e","Type":"ContainerStarted","Data":"cd7d6f04c08a6877470b0a85cfcbcc30613225de2226dd251bc1a0851f88a9de"}
Sep 18 19:38:29.968178 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:29.968153    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal" event={"ID":"3fba69533f2373986d0e5941a0f2828d","Type":"ContainerStarted","Data":"b3f79c25b2e602e13680ca69036f61211932067bc4477186dfa19db05344d5ae"}
Sep 18 19:38:30.106540 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.106453    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:30.107506 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.107480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bb2kg"
Sep 18 19:38:30.119431 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.119403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5x6hj"
Sep 18 19:38:30.126151 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.126129    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:30.134989 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.134970    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5xwm"
Sep 18 19:38:30.146911 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.146876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:30.157642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.157622    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fx5vm"
Sep 18 19:38:30.164405 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.164388    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls"
Sep 18 19:38:30.170235 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.170220    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spvxq"
Sep 18 19:38:30.179808 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.179788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w4rdh"
Sep 18 19:38:30.449283 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.449208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:30.449283 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.449260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:30.449505 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.449390    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:30.449505 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.449401    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:30.449505 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.449455    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs podName:2f45aa15-f96d-4fc2-8c7b-4e96873d6147 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:31.449441486 +0000 UTC m=+5.703648068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs") pod "network-metrics-daemon-q65cx" (UID: "2f45aa15-f96d-4fc2-8c7b-4e96873d6147") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:30.449505 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.449474    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret podName:4fd05598-f32e-42b9-82a7-cb695c62d50c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:31.449464075 +0000 UTC m=+5.703670660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret") pod "global-pull-secret-syncer-pkqrg" (UID: "4fd05598-f32e-42b9-82a7-cb695c62d50c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:30.550290 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.550250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:30.550451 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.550409    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:30.550451 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.550433    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:30.550451 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.550442    2583 projected.go:196] Error preparing data for projected volume kube-api-access-48cql for pod openshift-network-diagnostics/network-check-target-52cq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.550581 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.550501    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql podName:8492f48b-11e8-4c35-88ea-d9eaf0f6de53 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:31.550482038 +0000 UTC m=+5.804688629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-48cql" (UniqueName: "kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql") pod "network-check-target-52cq7" (UID: "8492f48b-11e8-4c35-88ea-d9eaf0f6de53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.848622 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.848570    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:29 +0000 UTC" deadline="2028-05-07 16:46:56.616352044 +0000 UTC"
Sep 18 19:38:30.848622 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.848618    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14325h8m25.767738388s"
Sep 18 19:38:30.960856 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.960826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:30.961015 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:30.960827    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:30.961015 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.960948    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:30.961015 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:30.961002    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:30.988657 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:30.988628    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3770c7_042c_40d6_954d_90f726d05bc2.slice/crio-e28134f624857057f88bc30035e10b4230c9b92390e3b30d5921f7b9764a338d WatchSource:0}: Error finding container e28134f624857057f88bc30035e10b4230c9b92390e3b30d5921f7b9764a338d: Status 404 returned error can't find the container with id e28134f624857057f88bc30035e10b4230c9b92390e3b30d5921f7b9764a338d
Sep 18 19:38:30.990373 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:30.990320    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd281f38_1f69_48e1_ac2a_f8b3cdd3b100.slice/crio-b3b4ffe357839ea93d781836dc9a80d409aebf33cf6b8fe2275a557bd6a97287 WatchSource:0}: Error finding container b3b4ffe357839ea93d781836dc9a80d409aebf33cf6b8fe2275a557bd6a97287: Status 404 returned error can't find the container with id b3b4ffe357839ea93d781836dc9a80d409aebf33cf6b8fe2275a557bd6a97287
Sep 18 19:38:30.991500 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:30.991415    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32f6f89_1b73_4dd5_82e5_7ca39ff3ee73.slice/crio-eef4e4164a837bfa84af3b09933ad067a4da3a8fdbab7546e5e00230823dbe3d WatchSource:0}: Error finding container eef4e4164a837bfa84af3b09933ad067a4da3a8fdbab7546e5e00230823dbe3d: Status 404 returned error can't find the container with id eef4e4164a837bfa84af3b09933ad067a4da3a8fdbab7546e5e00230823dbe3d
Sep 18 19:38:30.993895 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:30.993871    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3356ecda_7f08_4f66_a08e_2cae194cc7a0.slice/crio-e7bd42dba0756b4a59d19bec5e0192bc7733862aaee95d965ba18538b3224670 WatchSource:0}: Error finding container e7bd42dba0756b4a59d19bec5e0192bc7733862aaee95d965ba18538b3224670: Status 404 returned error can't find the container with id e7bd42dba0756b4a59d19bec5e0192bc7733862aaee95d965ba18538b3224670
Sep 18 19:38:30.998799 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:30.997777    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e640a31_2d0a_4a6f_ae71_f3df042960f8.slice/crio-db74203a0715457f933e169f80a360f7dcb90c7c5afd09d2c94700b15e7df2ec WatchSource:0}: Error finding container db74203a0715457f933e169f80a360f7dcb90c7c5afd09d2c94700b15e7df2ec: Status 404 returned error can't find the container with id db74203a0715457f933e169f80a360f7dcb90c7c5afd09d2c94700b15e7df2ec
Sep 18 19:38:31.000778 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:31.000675    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef6b572_90dc_4447_bc19_93f0367e9926.slice/crio-a43c612226b6f2ee0431fc1ef3564680e4a931d1a3cd1c1bbd60e258a89db016 WatchSource:0}: Error finding container a43c612226b6f2ee0431fc1ef3564680e4a931d1a3cd1c1bbd60e258a89db016: Status 404 returned error can't find the container with id a43c612226b6f2ee0431fc1ef3564680e4a931d1a3cd1c1bbd60e258a89db016
Sep 18 19:38:31.001656 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:31.001498    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116348dd_64ad_47d4_a5a8_b41540fd7a67.slice/crio-6eaa6af87e6a379b3198b15ae0ae72a8f632b13de47b50c8049bc1a7e2a2cf18 WatchSource:0}: Error finding container 6eaa6af87e6a379b3198b15ae0ae72a8f632b13de47b50c8049bc1a7e2a2cf18: Status 404 returned error can't find the container with id 6eaa6af87e6a379b3198b15ae0ae72a8f632b13de47b50c8049bc1a7e2a2cf18
Sep 18 19:38:31.004498 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:38:31.002284    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077e0f30_ad3e_44a5_bf8d_4cf88903e1b0.slice/crio-7826520957af8016372228d84d4fd85a103dd48edba977a1f8f7f7e9848823fc WatchSource:0}: Error finding container 7826520957af8016372228d84d4fd85a103dd48edba977a1f8f7f7e9848823fc: Status 404 returned error can't find the container with id 7826520957af8016372228d84d4fd85a103dd48edba977a1f8f7f7e9848823fc
Sep 18 19:38:31.458616 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.457615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:31.458616 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.457884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:31.458616 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.458024    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:31.458616 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.458086    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs podName:2f45aa15-f96d-4fc2-8c7b-4e96873d6147 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.458066471 +0000 UTC m=+7.712273059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs") pod "network-metrics-daemon-q65cx" (UID: "2f45aa15-f96d-4fc2-8c7b-4e96873d6147") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:31.458616 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.458487    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:31.458616 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.458537    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret podName:4fd05598-f32e-42b9-82a7-cb695c62d50c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.458522563 +0000 UTC m=+7.712729150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret") pod "global-pull-secret-syncer-pkqrg" (UID: "4fd05598-f32e-42b9-82a7-cb695c62d50c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:31.558973 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.558408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:31.558973 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.558556    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:31.558973 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.558573    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:31.558973 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.558585    2583 projected.go:196] Error preparing data for projected volume kube-api-access-48cql for pod openshift-network-diagnostics/network-check-target-52cq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:31.558973 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.558640    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql podName:8492f48b-11e8-4c35-88ea-d9eaf0f6de53 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.558622093 +0000 UTC m=+7.812828681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-48cql" (UniqueName: "kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql") pod "network-check-target-52cq7" (UID: "8492f48b-11e8-4c35-88ea-d9eaf0f6de53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:31.849049 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.848987    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:29 +0000 UTC" deadline="2028-03-08 03:36:56.320008438 +0000 UTC"
Sep 18 19:38:31.849049 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.849047    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12871h58m24.470965909s"
Sep 18 19:38:31.962167 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.961693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:31.962167 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:31.961821    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:31.981877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.981847    2583 generic.go:356] "Generic (PLEG): container finished" podID="62a84dcb18dfedc7e4fcb971d5de5a7e" containerID="7a33f5be9b2f0909dfb7732266639f81806875107e9fba7a2ea9f0dcaea29a67" exitCode=0
Sep 18 19:38:31.982004 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.981931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal" event={"ID":"62a84dcb18dfedc7e4fcb971d5de5a7e","Type":"ContainerDied","Data":"7a33f5be9b2f0909dfb7732266639f81806875107e9fba7a2ea9f0dcaea29a67"}
Sep 18 19:38:31.990398 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:31.990373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal" event={"ID":"3fba69533f2373986d0e5941a0f2828d","Type":"ContainerStarted","Data":"d58ca0a77df7a89f7c69af6b7d86ef0ad91710fdceb6de26fa3f59566d809fde"}
Sep 18 19:38:32.006163 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.006136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"6eaa6af87e6a379b3198b15ae0ae72a8f632b13de47b50c8049bc1a7e2a2cf18"}
Sep 18 19:38:32.017074 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.017013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spvxq" event={"ID":"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0","Type":"ContainerStarted","Data":"7826520957af8016372228d84d4fd85a103dd48edba977a1f8f7f7e9848823fc"}
Sep 18 19:38:32.024538 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.024294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5x6hj" event={"ID":"8e640a31-2d0a-4a6f-ae71-f3df042960f8","Type":"ContainerStarted","Data":"db74203a0715457f933e169f80a360f7dcb90c7c5afd09d2c94700b15e7df2ec"}
Sep 18 19:38:32.037385 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.037246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n9s2s" event={"ID":"3356ecda-7f08-4f66-a08e-2cae194cc7a0","Type":"ContainerStarted","Data":"e7bd42dba0756b4a59d19bec5e0192bc7733862aaee95d965ba18538b3224670"}
Sep 18 19:38:32.047562 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.047536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bb2kg" event={"ID":"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100","Type":"ContainerStarted","Data":"b3b4ffe357839ea93d781836dc9a80d409aebf33cf6b8fe2275a557bd6a97287"}
Sep 18 19:38:32.056198 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.056132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerStarted","Data":"e28134f624857057f88bc30035e10b4230c9b92390e3b30d5921f7b9764a338d"}
Sep 18 19:38:32.069482 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.069454    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fx5vm" event={"ID":"a3f524e7-7958-47df-8762-0839d84b63fd","Type":"ContainerStarted","Data":"1420f71781914ef57764a95b94f441bfd36bf234ca3902d8be9366de003788ba"}
Sep 18 19:38:32.086007 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.085971    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls" event={"ID":"5ef6b572-90dc-4447-bc19-93f0367e9926","Type":"ContainerStarted","Data":"a43c612226b6f2ee0431fc1ef3564680e4a931d1a3cd1c1bbd60e258a89db016"}
Sep 18 19:38:32.107607 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.107516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5xwm" event={"ID":"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73","Type":"ContainerStarted","Data":"eef4e4164a837bfa84af3b09933ad067a4da3a8fdbab7546e5e00230823dbe3d"}
Sep 18 19:38:32.961617 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.961585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:32.962044 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:32.961717    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:32.962176 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:32.962159    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:32.962311 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:32.962291    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:33.148805 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:33.148769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal" event={"ID":"62a84dcb18dfedc7e4fcb971d5de5a7e","Type":"ContainerStarted","Data":"4a2fd3c618fcafa972577eaa19ab100613904229c5f0a477195714de58821972"}
Sep 18 19:38:33.163589 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:33.163532    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-25.ec2.internal" podStartSLOduration=4.163513878 podStartE2EDuration="4.163513878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:32.014785854 +0000 UTC m=+6.268992459" watchObservedRunningTime="2026-09-18 19:38:33.163513878 +0000 UTC m=+7.417720479"
Sep 18 19:38:33.475824 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:33.475789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:33.475985 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:33.475879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:33.476060 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.476013    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.476116 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.476073    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret podName:4fd05598-f32e-42b9-82a7-cb695c62d50c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:37.476055623 +0000 UTC m=+11.730262219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret") pod "global-pull-secret-syncer-pkqrg" (UID: "4fd05598-f32e-42b9-82a7-cb695c62d50c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.476538 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.476513    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:33.476627 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.476572    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs podName:2f45aa15-f96d-4fc2-8c7b-4e96873d6147 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:37.476556142 +0000 UTC m=+11.730762737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs") pod "network-metrics-daemon-q65cx" (UID: "2f45aa15-f96d-4fc2-8c7b-4e96873d6147") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:33.576432 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:33.576397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:33.576593 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.576564    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:33.576593 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.576583    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:33.576593 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.576595    2583 projected.go:196] Error preparing data for projected volume kube-api-access-48cql for pod openshift-network-diagnostics/network-check-target-52cq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:33.576763 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.576658    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql podName:8492f48b-11e8-4c35-88ea-d9eaf0f6de53 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:37.576632974 +0000 UTC m=+11.830839573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-48cql" (UniqueName: "kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql") pod "network-check-target-52cq7" (UID: "8492f48b-11e8-4c35-88ea-d9eaf0f6de53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:33.964227 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:33.963684    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:33.964227 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:33.963809    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:34.961518 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:34.961489    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:34.961697 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:34.961612    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:34.961697 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:34.961676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:34.961795 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:34.961779    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:35.962453 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:35.962417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:35.962839 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:35.962572    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:36.961415 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:36.961380    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:36.961593 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:36.961534    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:36.961875 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:36.961849    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:36.961998 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:36.961939    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:37.507080 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:37.507036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:37.507616 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:37.507123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:37.507616 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.507226    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:37.507616 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.507282    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret podName:4fd05598-f32e-42b9-82a7-cb695c62d50c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:45.507263138 +0000 UTC m=+19.761469738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret") pod "global-pull-secret-syncer-pkqrg" (UID: "4fd05598-f32e-42b9-82a7-cb695c62d50c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:37.507790 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.507681    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:37.507790 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.507739    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs podName:2f45aa15-f96d-4fc2-8c7b-4e96873d6147 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:45.507716243 +0000 UTC m=+19.761922844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs") pod "network-metrics-daemon-q65cx" (UID: "2f45aa15-f96d-4fc2-8c7b-4e96873d6147") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:37.608194 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:37.607430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:37.608194 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.607596    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:37.608194 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.607610    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:37.608194 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.607619    2583 projected.go:196] Error preparing data for projected volume kube-api-access-48cql for pod openshift-network-diagnostics/network-check-target-52cq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:37.608194 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.607665    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql podName:8492f48b-11e8-4c35-88ea-d9eaf0f6de53 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:45.607649171 +0000 UTC m=+19.861855754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-48cql" (UniqueName: "kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql") pod "network-check-target-52cq7" (UID: "8492f48b-11e8-4c35-88ea-d9eaf0f6de53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:37.962083 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:37.962047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:37.962267 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:37.962147    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:38.961558 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:38.961163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:38.961558 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:38.961168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:38.961558 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:38.961276    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:38.961558 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:38.961378    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:39.961383 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:39.960805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:39.961383 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:39.960937    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:40.960934 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:40.960898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:40.961347 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:40.960902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:40.961347 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:40.961023    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:40.961347 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:40.961103    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:41.961643 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:41.961606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:41.962096 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:41.961742    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:42.961520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:42.961484    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:42.961679 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:42.961527    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:42.961679 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:42.961608    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:42.961993 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:42.961735    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:43.961430 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:43.961401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:43.961584 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:43.961529    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:44.960981 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:44.960899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:44.960981 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:44.960921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:44.961492 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:44.961028    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:44.961492 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:44.961169    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:45.559286 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:45.559243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:45.559464 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:45.559315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:45.559464 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.559419    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:45.559464 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.559425    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:45.559606 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.559470    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret podName:4fd05598-f32e-42b9-82a7-cb695c62d50c nodeName:}" failed. No retries permitted until 2026-09-18 19:39:01.559457606 +0000 UTC m=+35.813664188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret") pod "global-pull-secret-syncer-pkqrg" (UID: "4fd05598-f32e-42b9-82a7-cb695c62d50c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:45.559606 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.559495    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs podName:2f45aa15-f96d-4fc2-8c7b-4e96873d6147 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:01.559475854 +0000 UTC m=+35.813682438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs") pod "network-metrics-daemon-q65cx" (UID: "2f45aa15-f96d-4fc2-8c7b-4e96873d6147") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:45.660432 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:45.660393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:45.660610 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.660582    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:45.660610 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.660610    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:45.660713 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.660623    2583 projected.go:196] Error preparing data for projected volume kube-api-access-48cql for pod openshift-network-diagnostics/network-check-target-52cq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:45.660713 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.660683    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql podName:8492f48b-11e8-4c35-88ea-d9eaf0f6de53 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:01.660662863 +0000 UTC m=+35.914869445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-48cql" (UniqueName: "kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql") pod "network-check-target-52cq7" (UID: "8492f48b-11e8-4c35-88ea-d9eaf0f6de53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:45.961156 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:45.961081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:45.961611 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:45.961200    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:46.961417 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:46.961385    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:46.961417 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:46.961410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:46.961926 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:46.961514    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:46.961926 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:46.961631    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:47.962168 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:47.962134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:47.962584 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:47.962277    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:48.960731 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:48.960704    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:48.961186 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:48.960833    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:48.961186 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:48.960909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:48.961186 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:48.960983    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:49.181122 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.180939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n9s2s" event={"ID":"3356ecda-7f08-4f66-a08e-2cae194cc7a0","Type":"ContainerStarted","Data":"6dcbbb27d8fabfd8dad7d7cace4e46ec2db7882e0deb8d30382c43430356055d"}
Sep 18 19:38:49.182478 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.182456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bb2kg" event={"ID":"fd281f38-1f69-48e1-ac2a-f8b3cdd3b100","Type":"ContainerStarted","Data":"6d8d1170b021cecc64ce7318082e7b625cd4b14a7828feac3456111a3a7e02ba"}
Sep 18 19:38:49.183872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.183853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerStarted","Data":"84e80d813faf903d8ba35230ac8b7495a2b77b723530b401fced7126b007e7df"}
Sep 18 19:38:49.185237 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.185207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fx5vm" event={"ID":"a3f524e7-7958-47df-8762-0839d84b63fd","Type":"ContainerStarted","Data":"9a671a1c8d83885150ad88ddab43a57f93328498dc9907666afcd6f7eff1f28d"}
Sep 18 19:38:49.186590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.186570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls" event={"ID":"5ef6b572-90dc-4447-bc19-93f0367e9926","Type":"ContainerStarted","Data":"2ad1df042678c4d59f2bbdeb29cadf6d30f2ce1a56292099f09a02089f3f1827"}
Sep 18 19:38:49.187920 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.187900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5xwm" event={"ID":"d32f6f89-1b73-4dd5-82e5-7ca39ff3ee73","Type":"ContainerStarted","Data":"94850d87dde0e1232d4780f775433244e4025a07a90fd60d186f4f82737c04f0"}
Sep 18 19:38:49.189396 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.189374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"3ac2e84667c083928daef420f4411cc97f4c13e2df1657a3c93214822a2d03b2"}
Sep 18 19:38:49.190547 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.190529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spvxq" event={"ID":"077e0f30-ad3e-44a5-bf8d-4cf88903e1b0","Type":"ContainerStarted","Data":"f714a23542ccd1ea6bd63f3bc9e4db543d427373570868b4ac4c0639a8b21c06"}
Sep 18 19:38:49.196802 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.196761    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n9s2s" podStartSLOduration=7.664011351 podStartE2EDuration="21.19674939s" totalImagesPullingTime="13.532738039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:49.196649911 +0000 UTC m=+23.450856515" watchObservedRunningTime="2026-09-18 19:38:49.19674939 +0000 UTC m=+23.450955993"
Sep 18 19:38:49.196993 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.196972    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-25.ec2.internal" podStartSLOduration=20.196964393 podStartE2EDuration="20.196964393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:33.16426209 +0000 UTC m=+7.418468694" watchObservedRunningTime="2026-09-18 19:38:49.196964393 +0000 UTC m=+23.451170996"
Sep 18 19:38:49.210217 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.210187    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-spvxq" podStartSLOduration=3.41711168 podStartE2EDuration="21.210175887s" totalImagesPullingTime="17.793064207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:49.210034438 +0000 UTC m=+23.464241041" watchObservedRunningTime="2026-09-18 19:38:49.210175887 +0000 UTC m=+23.464382491"
Sep 18 19:38:49.250805 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.250767    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bb2kg" podStartSLOduration=3.444487879 podStartE2EDuration="21.250751109s" totalImagesPullingTime="17.80626323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:49.250705754 +0000 UTC m=+23.504912361" watchObservedRunningTime="2026-09-18 19:38:49.250751109 +0000 UTC m=+23.504957711"
Sep 18 19:38:49.264008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.263868    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fx5vm" podStartSLOduration=3.472443799 podStartE2EDuration="21.263848707s" totalImagesPullingTime="17.791404908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:49.263770386 +0000 UTC m=+23.517976991" watchObservedRunningTime="2026-09-18 19:38:49.263848707 +0000 UTC m=+23.518055314"
Sep 18 19:38:49.285936 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.285898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s5xwm" podStartSLOduration=3.440290794 podStartE2EDuration="21.28588509s" totalImagesPullingTime="17.845594296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:49.285546776 +0000 UTC m=+23.539753380" watchObservedRunningTime="2026-09-18 19:38:49.28588509 +0000 UTC m=+23.540091694"
Sep 18 19:38:49.799564 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.799391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:49.800142 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.800124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:49.961489 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:49.961463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:49.961632 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:49.961563    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:50.194898 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.194875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:38:50.195475 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.195184    2583 generic.go:356] "Generic (PLEG): container finished" podID="116348dd-64ad-47d4-a5a8-b41540fd7a67" containerID="13f8a616a93ec3abc2c27bb72af3c14e17f08dcdacdf89100c4b04330ad9b66f" exitCode=1
Sep 18 19:38:50.195475 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.195248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"f1f28db699c22f3f967a004cbafd13f4fc6b437627d68989f161a2046514e5e1"}
Sep 18 19:38:50.195475 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.195310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"b8b2240ff442e0447b84bd9b0eba93c466169a55b335101d7340bb906a4fc003"}
Sep 18 19:38:50.195475 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.195348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"42c42d619648f7adf9027ce0f32929d7637257703bc8f4bbb2eb472b3fd54d8e"}
Sep 18 19:38:50.195475 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.195392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"686c8f4bbce01649625cab2101a1824a3b3bb3391b4cea6819d538f8b679a7d5"}
Sep 18 19:38:50.195475 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.195418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerDied","Data":"13f8a616a93ec3abc2c27bb72af3c14e17f08dcdacdf89100c4b04330ad9b66f"}
Sep 18 19:38:50.196658 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.196629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5x6hj" event={"ID":"8e640a31-2d0a-4a6f-ae71-f3df042960f8","Type":"ContainerStarted","Data":"e6a7e7ee6d2f50e07cc9a6eef780ceb5ef92fdbae5d8173efa894824ee93e33f"}
Sep 18 19:38:50.198101 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.198075    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a3770c7-042c-40d6-954d-90f726d05bc2" containerID="84e80d813faf903d8ba35230ac8b7495a2b77b723530b401fced7126b007e7df" exitCode=0
Sep 18 19:38:50.198195 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.198170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerDied","Data":"84e80d813faf903d8ba35230ac8b7495a2b77b723530b401fced7126b007e7df"}
Sep 18 19:38:50.211727 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.211669    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5x6hj" podStartSLOduration=4.416962785 podStartE2EDuration="22.211655205s" totalImagesPullingTime="17.79469242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:50.211419209 +0000 UTC m=+24.465625813" watchObservedRunningTime="2026-09-18 19:38:50.211655205 +0000 UTC m=+24.465861808"
Sep 18 19:38:50.487114 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.487094    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:38:50.903766 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.903643    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:38:50.487109922Z","UUID":"aa4828ae-3f3a-41d7-a307-0005c2eff205","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:38:50.905197 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.905175    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:38:50.905311 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.905210    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:38:50.961455 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.960917    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:50.961455 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:50.961038    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:50.961455 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:50.960917    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:50.961760 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:50.961486    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:51.203987 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:51.203918    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:51.204477 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:51.204455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls" event={"ID":"5ef6b572-90dc-4447-bc19-93f0367e9926","Type":"ContainerStarted","Data":"ddae9e1ef79017babe4891c56772b8c75a0c7652f089c6455aea1af16125a30e"}
Sep 18 19:38:51.961501 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:51.961227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:51.961676 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:51.961651    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:52.209756 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:52.209717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls" event={"ID":"5ef6b572-90dc-4447-bc19-93f0367e9926","Type":"ContainerStarted","Data":"b23d3b86edfa23aaf07e0f4a14b92023651ba199d1f9be93838b935d4fa8d3f8"}
Sep 18 19:38:52.213487 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:52.213461    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:38:52.213783 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:52.213757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"18a17404b7aa05311cf45857c0add9933f530d0de44e296d09ded81126e8a023"}
Sep 18 19:38:52.227669 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:52.227620    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8qls" podStartSLOduration=3.916328368 podStartE2EDuration="24.227603852s" totalImagesPullingTime="20.311275484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:52.227370765 +0000 UTC m=+26.481577361" watchObservedRunningTime="2026-09-18 19:38:52.227603852 +0000 UTC m=+26.481810459"
Sep 18 19:38:52.961329 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:52.960890    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:52.961329 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:52.961006    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:52.961329 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:52.961050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:52.961329 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:52.961165    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:53.961224 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:53.961199    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:53.961617 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:53.961333    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:54.961840 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:54.961636    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:54.962375 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:54.961678    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:54.962375 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:54.961878    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:54.962375 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:54.961975    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:55.223505 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.223432    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:38:55.223840 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.223815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"94edd743320761b6772c4d3a885750f94b56294ff7dfaa07f95e32596c7a6b67"}
Sep 18 19:38:55.224142 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.224108    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:55.224261 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.224215    2583 scope.go:117] "RemoveContainer" containerID="13f8a616a93ec3abc2c27bb72af3c14e17f08dcdacdf89100c4b04330ad9b66f"
Sep 18 19:38:55.225838 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.225813    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a3770c7-042c-40d6-954d-90f726d05bc2" containerID="a10a7c312f58cd7c4789db94a31167ef34c5b0dd3435b729a8b20556763946b3" exitCode=0
Sep 18 19:38:55.225939 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.225858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerDied","Data":"a10a7c312f58cd7c4789db94a31167ef34c5b0dd3435b729a8b20556763946b3"}
Sep 18 19:38:55.240581 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.240543    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:55.961446 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:55.961415    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:55.961640 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:55.961534    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:56.232235 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.232175    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:38:56.232585 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.232559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" event={"ID":"116348dd-64ad-47d4-a5a8-b41540fd7a67","Type":"ContainerStarted","Data":"f3d9b83cf45cff0e3cb93ab5d3dc3f67f1d813ec8393b5e4d754f89ce5c14732"}
Sep 18 19:38:56.232689 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.232674    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:56.232942 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.232917    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:56.248331 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.248313    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:56.262596 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.262554    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" podStartSLOduration=10.44556176 podStartE2EDuration="28.262542027s" totalImagesPullingTime="17.816980267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:56.262263061 +0000 UTC m=+30.516469667" watchObservedRunningTime="2026-09-18 19:38:56.262542027 +0000 UTC m=+30.516748630"
Sep 18 19:38:56.346425 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.346400    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-52cq7"]
Sep 18 19:38:56.346546 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.346507    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:56.346607 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:56.346590    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:56.346659 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.346618    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pkqrg"]
Sep 18 19:38:56.346708 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.346695    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:56.346795 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:56.346773    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:56.348767 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.348746    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q65cx"]
Sep 18 19:38:56.348886 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:56.348833    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:56.348943 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:56.348921    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:57.236843 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:57.236662    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a3770c7-042c-40d6-954d-90f726d05bc2" containerID="9e7a29e702496820bfc21374af845e342f53e5a5f9f083db3ca09e07ea576385" exitCode=0
Sep 18 19:38:57.237234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:57.236748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerDied","Data":"9e7a29e702496820bfc21374af845e342f53e5a5f9f083db3ca09e07ea576385"}
Sep 18 19:38:57.237234 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:57.236975    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:57.444462 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:57.444433    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:38:57.962608 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:57.962562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:57.962804 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:57.962677    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:57.962804 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:57.962696    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:57.962804 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:57.962730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:57.962964 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:57.962806    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:57.962964 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:57.962860    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:38:59.243236 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.243203    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a3770c7-042c-40d6-954d-90f726d05bc2" containerID="df7e6bed28e7c4fcba78e11941f19658815c72ceee96b2efe1075fac1cca5f1e" exitCode=0
Sep 18 19:38:59.243920 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.243284    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerDied","Data":"df7e6bed28e7c4fcba78e11941f19658815c72ceee96b2efe1075fac1cca5f1e"}
Sep 18 19:38:59.263428 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.263393    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s" podUID="116348dd-64ad-47d4-a5a8-b41540fd7a67" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 18 19:38:59.873864 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.873831    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:59.874030 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.873954    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:59.874983 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.874956    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n9s2s"
Sep 18 19:38:59.960982 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.960956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:38:59.960982 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.960988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:38:59.961217 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:38:59.960956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:38:59.961217 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:59.961080    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q65cx" podUID="2f45aa15-f96d-4fc2-8c7b-4e96873d6147"
Sep 18 19:38:59.961217 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:59.961158    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkqrg" podUID="4fd05598-f32e-42b9-82a7-cb695c62d50c"
Sep 18 19:38:59.961393 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:38:59.961220    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-52cq7" podUID="8492f48b-11e8-4c35-88ea-d9eaf0f6de53"
Sep 18 19:39:01.572623 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.572587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:39:01.573204 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.572673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:39:01.573204 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.572719    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:01.573204 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.572775    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:01.573204 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.572796    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs podName:2f45aa15-f96d-4fc2-8c7b-4e96873d6147 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:33.572774869 +0000 UTC m=+67.826981465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs") pod "network-metrics-daemon-q65cx" (UID: "2f45aa15-f96d-4fc2-8c7b-4e96873d6147") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:01.573204 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.572816    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret podName:4fd05598-f32e-42b9-82a7-cb695c62d50c nodeName:}" failed. No retries permitted until 2026-09-18 19:39:33.572804385 +0000 UTC m=+67.827010983 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret") pod "global-pull-secret-syncer-pkqrg" (UID: "4fd05598-f32e-42b9-82a7-cb695c62d50c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:01.673055 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.672995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:39:01.673243 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.673163    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:01.673243 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.673189    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:01.673243 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.673199    2583 projected.go:196] Error preparing data for projected volume kube-api-access-48cql for pod openshift-network-diagnostics/network-check-target-52cq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:01.673373 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.673250    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql podName:8492f48b-11e8-4c35-88ea-d9eaf0f6de53 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:33.67323564 +0000 UTC m=+67.927442221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-48cql" (UniqueName: "kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql") pod "network-check-target-52cq7" (UID: "8492f48b-11e8-4c35-88ea-d9eaf0f6de53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:01.681991 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.681971    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-25.ec2.internal" event="NodeReady"
Sep 18 19:39:01.682132 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.682103    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:39:01.727874 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.727844    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wbtwc"]
Sep 18 19:39:01.742904 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.742875    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-shv9m"]
Sep 18 19:39:01.743670 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.743642    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.746099 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.746075    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:39:01.746205 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.746129    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:39:01.746205 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.746133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5f8sq\""
Sep 18 19:39:01.757173 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.757157    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wbtwc"]
Sep 18 19:39:01.757294 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.757187    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shv9m"]
Sep 18 19:39:01.757294 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.757284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:01.759949 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.759835    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:39:01.759949 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.759860    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:39:01.759949 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.759889    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q4x2f\""
Sep 18 19:39:01.760179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.760000    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:39:01.874497 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.874415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1044dfff-ccf8-4561-b87c-c5fc588f6603-config-volume\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.874497 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.874462    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2k8p\" (UniqueName: \"kubernetes.io/projected/2e30c6ab-20fd-40dd-952a-9dfe7562d4cb-kube-api-access-s2k8p\") pod \"ingress-canary-shv9m\" (UID: \"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb\") " pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:01.874754 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.874497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.874754 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.874590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1044dfff-ccf8-4561-b87c-c5fc588f6603-tmp-dir\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.874754 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.874625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcw4n\" (UniqueName: \"kubernetes.io/projected/1044dfff-ccf8-4561-b87c-c5fc588f6603-kube-api-access-rcw4n\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.874754 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.874652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e30c6ab-20fd-40dd-952a-9dfe7562d4cb-cert\") pod \"ingress-canary-shv9m\" (UID: \"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb\") " pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:01.961414 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.961379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:39:01.961625 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.961423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:39:01.961625 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.961385    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:39:01.963632 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.963613    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:01.964088 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.964069    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c9m8\""
Sep 18 19:39:01.964334 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.964317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pwxm5\""
Sep 18 19:39:01.964427 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.964414    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:01.964613 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.964597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:01.964825 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.964808    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:01.975316 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.975294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2k8p\" (UniqueName: \"kubernetes.io/projected/2e30c6ab-20fd-40dd-952a-9dfe7562d4cb-kube-api-access-s2k8p\") pod \"ingress-canary-shv9m\" (UID: \"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb\") " pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:01.975433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.975347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.975433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.975417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1044dfff-ccf8-4561-b87c-c5fc588f6603-tmp-dir\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.975642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.975444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcw4n\" (UniqueName: \"kubernetes.io/projected/1044dfff-ccf8-4561-b87c-c5fc588f6603-kube-api-access-rcw4n\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.975642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.975470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e30c6ab-20fd-40dd-952a-9dfe7562d4cb-cert\") pod \"ingress-canary-shv9m\" (UID: \"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb\") " pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:01.975642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.975531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1044dfff-ccf8-4561-b87c-c5fc588f6603-config-volume\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.976121 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.976106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1044dfff-ccf8-4561-b87c-c5fc588f6603-config-volume\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.976393 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.976378    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:01.976454 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:01.976433    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls podName:1044dfff-ccf8-4561-b87c-c5fc588f6603 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.476415806 +0000 UTC m=+36.730622412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls") pod "dns-default-wbtwc" (UID: "1044dfff-ccf8-4561-b87c-c5fc588f6603") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:01.976811 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.976791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1044dfff-ccf8-4561-b87c-c5fc588f6603-tmp-dir\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:01.981775 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.981754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e30c6ab-20fd-40dd-952a-9dfe7562d4cb-cert\") pod \"ingress-canary-shv9m\" (UID: \"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb\") " pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:01.984509 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.984489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2k8p\" (UniqueName: \"kubernetes.io/projected/2e30c6ab-20fd-40dd-952a-9dfe7562d4cb-kube-api-access-s2k8p\") pod \"ingress-canary-shv9m\" (UID: \"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb\") " pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:01.985104 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:01.985088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcw4n\" (UniqueName: \"kubernetes.io/projected/1044dfff-ccf8-4561-b87c-c5fc588f6603-kube-api-access-rcw4n\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:02.050647 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.050624    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dng7g"]
Sep 18 19:39:02.063058 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.063036    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.065765 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.065740    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:39:02.067036 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.066276    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:39:02.067036 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.066347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shv9m"
Sep 18 19:39:02.067036 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.065767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:39:02.067231 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.067036    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:39:02.070318 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.067641    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-696cc\""
Sep 18 19:39:02.070318 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.069022    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dng7g"]
Sep 18 19:39:02.176922 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.176853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zgp\" (UniqueName: \"kubernetes.io/projected/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-api-access-l6zgp\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.177091 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.176926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.177091 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.176986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.177091 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.177056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e92495-9fea-41e7-8ec2-fdfb24825356-data-volume\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.177211 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.177125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30e92495-9fea-41e7-8ec2-fdfb24825356-crio-socket\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.278413 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.278380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.278634 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.278464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e92495-9fea-41e7-8ec2-fdfb24825356-data-volume\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.278634 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.278499    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.278634 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.278569    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.778553385 +0000 UTC m=+37.032759971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.278817 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.278671    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30e92495-9fea-41e7-8ec2-fdfb24825356-crio-socket\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.278817 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.278505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30e92495-9fea-41e7-8ec2-fdfb24825356-crio-socket\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.278817 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.278732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zgp\" (UniqueName: \"kubernetes.io/projected/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-api-access-l6zgp\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.278817 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.278801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.278817 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.278808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e92495-9fea-41e7-8ec2-fdfb24825356-data-volume\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.279041 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.278945    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.279041 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.279003    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.778990574 +0000 UTC m=+37.033197160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.288954 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.288930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zgp\" (UniqueName: \"kubernetes.io/projected/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-api-access-l6zgp\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.480932 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.480845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:02.481165 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.481000    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:02.481165 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.481064    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls podName:1044dfff-ccf8-4561-b87c-c5fc588f6603 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.481046394 +0000 UTC m=+37.735252993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls") pod "dns-default-wbtwc" (UID: "1044dfff-ccf8-4561-b87c-c5fc588f6603") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:02.783577 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.783371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.784051 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.783609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:02.784051 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.783522    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.784051 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.783702    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.783687507 +0000 UTC m=+38.037894089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.784051 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.783744    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.784051 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:02.783807    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.783789516 +0000 UTC m=+38.037996106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.921553 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.921524    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"]
Sep 18 19:39:02.950124 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.950096    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"]
Sep 18 19:39:02.950300 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.950217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"
Sep 18 19:39:02.952708 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.952688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b7jzs\""
Sep 18 19:39:02.953020 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.953003    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 19:39:02.953020 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:02.953011    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 19:39:03.086963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.086869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhdz\" (UniqueName: \"kubernetes.io/projected/1e14754c-369c-43a1-86c2-c89a4594a94a-kube-api-access-kfhdz\") pod \"migrator-76794cbd6c-wsbmd\" (UID: \"1e14754c-369c-43a1-86c2-c89a4594a94a\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"
Sep 18 19:39:03.187985 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.187950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhdz\" (UniqueName: \"kubernetes.io/projected/1e14754c-369c-43a1-86c2-c89a4594a94a-kube-api-access-kfhdz\") pod \"migrator-76794cbd6c-wsbmd\" (UID: \"1e14754c-369c-43a1-86c2-c89a4594a94a\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"
Sep 18 19:39:03.196536 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.196499    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhdz\" (UniqueName: \"kubernetes.io/projected/1e14754c-369c-43a1-86c2-c89a4594a94a-kube-api-access-kfhdz\") pod \"migrator-76794cbd6c-wsbmd\" (UID: \"1e14754c-369c-43a1-86c2-c89a4594a94a\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"
Sep 18 19:39:03.260454 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.260429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"
Sep 18 19:39:03.489804 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.489733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:03.489953 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:03.489869    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:03.489953 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:03.489931    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls podName:1044dfff-ccf8-4561-b87c-c5fc588f6603 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.489916666 +0000 UTC m=+39.744123250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls") pod "dns-default-wbtwc" (UID: "1044dfff-ccf8-4561-b87c-c5fc588f6603") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:03.760596 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.760567    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-7hb8w"]
Sep 18 19:39:03.778239 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.778214    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-7hb8w"]
Sep 18 19:39:03.778380 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.778243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.780647 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.780630    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 19:39:03.780971 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.780951    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 19:39:03.781048 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.780959    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 19:39:03.781645 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.781627    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 19:39:03.781645 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.781642    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h6nqx\""
Sep 18 19:39:03.791790 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.791774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:03.792047 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.791805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:03.792047 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:03.791896    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:03.792047 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:03.791917    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:03.792047 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:03.791949    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.791932759 +0000 UTC m=+40.046139357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:03.792047 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:03.791974    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.791962001 +0000 UTC m=+40.046168584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:03.888986 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.888965    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shv9m"]
Sep 18 19:39:03.889074 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.889021    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd"]
Sep 18 19:39:03.892105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.892080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-signing-cabundle\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.892233 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.892141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-signing-key\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.892233 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.892199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn85z\" (UniqueName: \"kubernetes.io/projected/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-kube-api-access-sn85z\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.892233 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:03.892224    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e30c6ab_20fd_40dd_952a_9dfe7562d4cb.slice/crio-fead3dd5a0d275a68b814fd04e560129c43745ba8c8de3a45d1c82430bc8cf22 WatchSource:0}: Error finding container fead3dd5a0d275a68b814fd04e560129c43745ba8c8de3a45d1c82430bc8cf22: Status 404 returned error can't find the container with id fead3dd5a0d275a68b814fd04e560129c43745ba8c8de3a45d1c82430bc8cf22
Sep 18 19:39:03.893069 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:03.893048    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e14754c_369c_43a1_86c2_c89a4594a94a.slice/crio-dc979828e944b29d0079e3af6b508e1490d739d3f8f932b89581da26cf4b5bf0 WatchSource:0}: Error finding container dc979828e944b29d0079e3af6b508e1490d739d3f8f932b89581da26cf4b5bf0: Status 404 returned error can't find the container with id dc979828e944b29d0079e3af6b508e1490d739d3f8f932b89581da26cf4b5bf0
Sep 18 19:39:03.992984 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.992960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-signing-cabundle\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.993098 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.993005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-signing-key\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.993098 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.993040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn85z\" (UniqueName: \"kubernetes.io/projected/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-kube-api-access-sn85z\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.993705 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.993677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-signing-cabundle\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:03.996575 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:03.996552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-signing-key\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:04.001083 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.001051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn85z\" (UniqueName: \"kubernetes.io/projected/2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c-kube-api-access-sn85z\") pod \"service-ca-55f6fc7b86-7hb8w\" (UID: \"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:04.022100 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.022037    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spvxq_077e0f30-ad3e-44a5-bf8d-4cf88903e1b0/dns-node-resolver/0.log"
Sep 18 19:39:04.087717 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.087639    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w"
Sep 18 19:39:04.259416 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.259384    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a3770c7-042c-40d6-954d-90f726d05bc2" containerID="af05bfba77e49fd34619fe157f4f35a0ff6dcf8886882795158ea8dcb1860d4c" exitCode=0
Sep 18 19:39:04.259582 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.259478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerDied","Data":"af05bfba77e49fd34619fe157f4f35a0ff6dcf8886882795158ea8dcb1860d4c"}
Sep 18 19:39:04.261623 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.261596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd" event={"ID":"1e14754c-369c-43a1-86c2-c89a4594a94a","Type":"ContainerStarted","Data":"dc979828e944b29d0079e3af6b508e1490d739d3f8f932b89581da26cf4b5bf0"}
Sep 18 19:39:04.262950 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.262916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shv9m" event={"ID":"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb","Type":"ContainerStarted","Data":"fead3dd5a0d275a68b814fd04e560129c43745ba8c8de3a45d1c82430bc8cf22"}
Sep 18 19:39:04.363425 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:04.362555    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-7hb8w"]
Sep 18 19:39:04.374342 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:04.374314    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6f0c20_b06b_4e7a_aa58_d6bb7a8a895c.slice/crio-e969305ef91689459fa3a513739b58eda63dedcbcbaa90b5411577aaffcf185e WatchSource:0}: Error finding container e969305ef91689459fa3a513739b58eda63dedcbcbaa90b5411577aaffcf185e: Status 404 returned error can't find the container with id e969305ef91689459fa3a513739b58eda63dedcbcbaa90b5411577aaffcf185e
Sep 18 19:39:05.222323 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.222289    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fx5vm_a3f524e7-7958-47df-8762-0839d84b63fd/node-ca/0.log"
Sep 18 19:39:05.271054 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.271020    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a3770c7-042c-40d6-954d-90f726d05bc2" containerID="52e51e5c5533c7cb2cd66a9a152c1afeb7160fea66ee0b8c5af4c5ed0a6ae409" exitCode=0
Sep 18 19:39:05.271221 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.271134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerDied","Data":"52e51e5c5533c7cb2cd66a9a152c1afeb7160fea66ee0b8c5af4c5ed0a6ae409"}
Sep 18 19:39:05.274847 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.274742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd" event={"ID":"1e14754c-369c-43a1-86c2-c89a4594a94a","Type":"ContainerStarted","Data":"c1256e2ca1d39261ffeff1045128338f6b2f535dc14e19d63d99d60af34b11f1"}
Sep 18 19:39:05.274945 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.274874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd" event={"ID":"1e14754c-369c-43a1-86c2-c89a4594a94a","Type":"ContainerStarted","Data":"40da2aa78d4817310e23f5f0bc6050ad20e6eb060e18c21eb33be2eccc746cbe"}
Sep 18 19:39:05.276318 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.276259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w" event={"ID":"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c","Type":"ContainerStarted","Data":"e969305ef91689459fa3a513739b58eda63dedcbcbaa90b5411577aaffcf185e"}
Sep 18 19:39:05.308442 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.308389    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wsbmd" podStartSLOduration=2.219866408 podStartE2EDuration="3.308369102s" totalImagesPullingTime="1.088502694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:05.30720417 +0000 UTC m=+39.561410786" watchObservedRunningTime="2026-09-18 19:39:05.308369102 +0000 UTC m=+39.562575708"
Sep 18 19:39:05.507414 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.507383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:05.507591 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:05.507526    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:05.507648 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:05.507601    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls podName:1044dfff-ccf8-4561-b87c-c5fc588f6603 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.507579067 +0000 UTC m=+43.761785669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls") pod "dns-default-wbtwc" (UID: "1044dfff-ccf8-4561-b87c-c5fc588f6603") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:05.809144 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.809100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:05.809332 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:05.809241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:05.809438 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:05.809383    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:05.809438 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:05.809427    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:05.809543 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:05.809445    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.809427239 +0000 UTC m=+44.063633838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:05.809543 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:05.809466    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.809456527 +0000 UTC m=+44.063663109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:06.283261 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:06.283222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" event={"ID":"0a3770c7-042c-40d6-954d-90f726d05bc2","Type":"ContainerStarted","Data":"fa31b49de1ae361d276caa09cbb8de6204f8cc5c6b9906599536b5642dbd3c07"}
Sep 18 19:39:06.308340 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:06.308289    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w4rdh" podStartSLOduration=9.382587493 podStartE2EDuration="38.308274291s" totalImagesPullingTime="28.925686798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:06.306606999 +0000 UTC m=+40.560813620" watchObservedRunningTime="2026-09-18 19:39:06.308274291 +0000 UTC m=+40.562480895"
Sep 18 19:39:07.287523 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:07.287484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shv9m" event={"ID":"2e30c6ab-20fd-40dd-952a-9dfe7562d4cb","Type":"ContainerStarted","Data":"f1437448f5913de2338d9f2ea844ebca1a8a733e965a90d131158ffdc13bfc12"}
Sep 18 19:39:07.289230 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:07.289205    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w" event={"ID":"2f6f0c20-b06b-4e7a-aa58-d6bb7a8a895c","Type":"ContainerStarted","Data":"322d5bc94a4016d3c73fc9bdb4e1d2ed3d5904ab5c4cf1008778c636efca775d"}
Sep 18 19:39:07.303880 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:07.303837    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-shv9m" podStartSLOduration=3.465378323 podStartE2EDuration="6.303825642s" totalImagesPullingTime="2.838447319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:07.302504385 +0000 UTC m=+41.556710991" watchObservedRunningTime="2026-09-18 19:39:07.303825642 +0000 UTC m=+41.558032243"
Sep 18 19:39:07.318522 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:07.318388    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-7hb8w" podStartSLOduration=1.961291854 podStartE2EDuration="4.318373065s" totalImagesPullingTime="2.357081211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:07.317864907 +0000 UTC m=+41.572071510" watchObservedRunningTime="2026-09-18 19:39:07.318373065 +0000 UTC m=+41.572579690"
Sep 18 19:39:09.537033 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:09.536995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:09.537484 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:09.537148    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:09.537484 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:09.537215    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls podName:1044dfff-ccf8-4561-b87c-c5fc588f6603 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.537198904 +0000 UTC m=+51.791405485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls") pod "dns-default-wbtwc" (UID: "1044dfff-ccf8-4561-b87c-c5fc588f6603") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:09.838904 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:09.838819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:09.838904 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:09.838868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:09.839087 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:09.838935    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.839087 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:09.838971    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.839087 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:09.838985    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.838969411 +0000 UTC m=+52.093175994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.839087 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:09.839073    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.839061178 +0000 UTC m=+52.093267760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:17.593826 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:17.593788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:17.596994 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:17.596975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1044dfff-ccf8-4561-b87c-c5fc588f6603-metrics-tls\") pod \"dns-default-wbtwc\" (UID: \"1044dfff-ccf8-4561-b87c-c5fc588f6603\") " pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:17.655062 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:17.655028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:17.896175 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:17.896081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:17.896175 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:17.896122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:17.896413 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:17.896230    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:17.896413 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:17.896292    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls podName:30e92495-9fea-41e7-8ec2-fdfb24825356 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:33.896274358 +0000 UTC m=+68.150480940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dng7g" (UID: "30e92495-9fea-41e7-8ec2-fdfb24825356") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:17.896792 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:17.896767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30e92495-9fea-41e7-8ec2-fdfb24825356-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:17.955541 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:17.955514    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wbtwc"]
Sep 18 19:39:17.959102 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:17.959076    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1044dfff_ccf8_4561_b87c_c5fc588f6603.slice/crio-2dcd16dedb4dcd683d4befcde202b6cf3e5119dac0801e1b5bb2d3830dbd2347 WatchSource:0}: Error finding container 2dcd16dedb4dcd683d4befcde202b6cf3e5119dac0801e1b5bb2d3830dbd2347: Status 404 returned error can't find the container with id 2dcd16dedb4dcd683d4befcde202b6cf3e5119dac0801e1b5bb2d3830dbd2347
Sep 18 19:39:18.318391 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:18.318341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbtwc" event={"ID":"1044dfff-ccf8-4561-b87c-c5fc588f6603","Type":"ContainerStarted","Data":"2dcd16dedb4dcd683d4befcde202b6cf3e5119dac0801e1b5bb2d3830dbd2347"}
Sep 18 19:39:20.326129 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:20.326092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbtwc" event={"ID":"1044dfff-ccf8-4561-b87c-c5fc588f6603","Type":"ContainerStarted","Data":"9469dc6d7fec9b485164e3c0a4ff46cc7be294fd3eb178853b7b75f4261dbe8b"}
Sep 18 19:39:20.326537 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:20.326186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbtwc" event={"ID":"1044dfff-ccf8-4561-b87c-c5fc588f6603","Type":"ContainerStarted","Data":"4bd05a9ac1b4d52c71a5901816d820d62f0149c47ef459ea620c5215bbebe601"}
Sep 18 19:39:20.326537 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:20.326268    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:20.344848 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:20.344806    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wbtwc" podStartSLOduration=17.329094828 podStartE2EDuration="19.344794607s" totalImagesPullingTime="2.015699779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:20.344517051 +0000 UTC m=+54.598723655" watchObservedRunningTime="2026-09-18 19:39:20.344794607 +0000 UTC m=+54.599001211"
Sep 18 19:39:26.212517 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.212484    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"]
Sep 18 19:39:26.245465 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.245434    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"]
Sep 18 19:39:26.245619 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.245534    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.248330 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.248308    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 19:39:26.248484 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.248332    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:39:26.248845 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.248825    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 19:39:26.248934 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.248827    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:39:26.248934 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.248861    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:39:26.249630 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.249602    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 19:39:26.249738 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.249612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 19:39:26.355476 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.355447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9q4x\" (UniqueName: \"kubernetes.io/projected/456f8c11-10f6-4985-86a3-fe81ab721e15-kube-api-access-z9q4x\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.355609 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.355509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-ca\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.355609 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.355543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.355609 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.355569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-hub\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.355737 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.355637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.355737 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.355656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/456f8c11-10f6-4985-86a3-fe81ab721e15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.456216 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.456189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.456216 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.456216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-hub\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.456384 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.456251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.456452 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.456398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/456f8c11-10f6-4985-86a3-fe81ab721e15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.456506 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.456494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9q4x\" (UniqueName: \"kubernetes.io/projected/456f8c11-10f6-4985-86a3-fe81ab721e15-kube-api-access-z9q4x\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.456559 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.456546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-ca\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.457329 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.457224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/456f8c11-10f6-4985-86a3-fe81ab721e15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.458899 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.458879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-ca\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.459061 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.459039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.459061 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.459052    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-hub\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.459149 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.459058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/456f8c11-10f6-4985-86a3-fe81ab721e15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.466179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.466131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9q4x\" (UniqueName: \"kubernetes.io/projected/456f8c11-10f6-4985-86a3-fe81ab721e15-kube-api-access-z9q4x\") pod \"cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb\" (UID: \"456f8c11-10f6-4985-86a3-fe81ab721e15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.566898 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.566880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"
Sep 18 19:39:26.857957 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:26.857935    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb"]
Sep 18 19:39:26.859991 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:26.859956    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod456f8c11_10f6_4985_86a3_fe81ab721e15.slice/crio-719441a373540eff40456d42ae5ac45d2048cab10fae71f7ad76c1b1496a2006 WatchSource:0}: Error finding container 719441a373540eff40456d42ae5ac45d2048cab10fae71f7ad76c1b1496a2006: Status 404 returned error can't find the container with id 719441a373540eff40456d42ae5ac45d2048cab10fae71f7ad76c1b1496a2006
Sep 18 19:39:27.349520 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:27.349482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb" event={"ID":"456f8c11-10f6-4985-86a3-fe81ab721e15","Type":"ContainerStarted","Data":"719441a373540eff40456d42ae5ac45d2048cab10fae71f7ad76c1b1496a2006"}
Sep 18 19:39:29.254813 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:29.254787    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzl5s"
Sep 18 19:39:30.332318 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:30.332298    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wbtwc"
Sep 18 19:39:30.360152 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:30.360125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb" event={"ID":"456f8c11-10f6-4985-86a3-fe81ab721e15","Type":"ContainerStarted","Data":"6a4c40851d3a699ed383b5781875016f9a57fb5a202eeb31d7c56e01c26c58ee"}
Sep 18 19:39:30.360253 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:30.360169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb" event={"ID":"456f8c11-10f6-4985-86a3-fe81ab721e15","Type":"ContainerStarted","Data":"e8211f6db2a7ca0ee3eafb17744abbe4d09d808445d19499ed4a59d217718ada"}
Sep 18 19:39:31.364746 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:31.364710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb" event={"ID":"456f8c11-10f6-4985-86a3-fe81ab721e15","Type":"ContainerStarted","Data":"80ed27d21c69c29a3e0d71543d4a571f9a4de27abcb515f593c6c0e11c838986"}
Sep 18 19:39:31.385168 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:31.385120    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fd9d94dbb-ljkxb" podStartSLOduration=1.998741817 podStartE2EDuration="5.385106276s" totalImagesPullingTime="3.386364459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:31.383706709 +0000 UTC m=+65.637913313" watchObservedRunningTime="2026-09-18 19:39:31.385106276 +0000 UTC m=+65.639312880"
Sep 18 19:39:33.609742 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.609710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:39:33.610150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.609758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:39:33.612070 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.612051    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:33.612123 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.612091    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:33.623049 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.623021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f45aa15-f96d-4fc2-8c7b-4e96873d6147-metrics-certs\") pod \"network-metrics-daemon-q65cx\" (UID: \"2f45aa15-f96d-4fc2-8c7b-4e96873d6147\") " pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:39:33.623049 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.623030    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd05598-f32e-42b9-82a7-cb695c62d50c-original-pull-secret\") pod \"global-pull-secret-syncer-pkqrg\" (UID: \"4fd05598-f32e-42b9-82a7-cb695c62d50c\") " pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:39:33.710065 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.710043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:39:33.712229 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.712214    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:33.722313 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.722289    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:33.733376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.733336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cql\" (UniqueName: \"kubernetes.io/projected/8492f48b-11e8-4c35-88ea-d9eaf0f6de53-kube-api-access-48cql\") pod \"network-check-target-52cq7\" (UID: \"8492f48b-11e8-4c35-88ea-d9eaf0f6de53\") " pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:39:33.774675 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.774648    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pwxm5\""
Sep 18 19:39:33.779532 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.779518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkqrg"
Sep 18 19:39:33.782693 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.782673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:39:33.795712 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.795690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c9m8\""
Sep 18 19:39:33.803658 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.803639    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q65cx"
Sep 18 19:39:33.912082 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.912059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:33.914227 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:33.914199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30e92495-9fea-41e7-8ec2-fdfb24825356-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dng7g\" (UID: \"30e92495-9fea-41e7-8ec2-fdfb24825356\") " pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:34.051267 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.051238    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pkqrg"]
Sep 18 19:39:34.054466 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.054442    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-52cq7"]
Sep 18 19:39:34.054804 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:34.054783    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd05598_f32e_42b9_82a7_cb695c62d50c.slice/crio-e57a1c84811ceee51360a5983659dd4bf14629a2f014587a4637744f07f244e1 WatchSource:0}: Error finding container e57a1c84811ceee51360a5983659dd4bf14629a2f014587a4637744f07f244e1: Status 404 returned error can't find the container with id e57a1c84811ceee51360a5983659dd4bf14629a2f014587a4637744f07f244e1
Sep 18 19:39:34.057468 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:34.057450    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8492f48b_11e8_4c35_88ea_d9eaf0f6de53.slice/crio-9b8ebec211a9da7913f817c43921ff5ab965a250f85064c9d7d812daae854473 WatchSource:0}: Error finding container 9b8ebec211a9da7913f817c43921ff5ab965a250f85064c9d7d812daae854473: Status 404 returned error can't find the container with id 9b8ebec211a9da7913f817c43921ff5ab965a250f85064c9d7d812daae854473
Sep 18 19:39:34.088462 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.088441    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q65cx"]
Sep 18 19:39:34.090526 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:34.090503    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f45aa15_f96d_4fc2_8c7b_4e96873d6147.slice/crio-102f772cf0e27d366f7d5336bcd0d61a1f2a093edebc62adfb0d89f279b2b7f1 WatchSource:0}: Error finding container 102f772cf0e27d366f7d5336bcd0d61a1f2a093edebc62adfb0d89f279b2b7f1: Status 404 returned error can't find the container with id 102f772cf0e27d366f7d5336bcd0d61a1f2a093edebc62adfb0d89f279b2b7f1
Sep 18 19:39:34.179762 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.179707    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-696cc\""
Sep 18 19:39:34.187892 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.187878    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dng7g"
Sep 18 19:39:34.379030 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.378997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q65cx" event={"ID":"2f45aa15-f96d-4fc2-8c7b-4e96873d6147","Type":"ContainerStarted","Data":"102f772cf0e27d366f7d5336bcd0d61a1f2a093edebc62adfb0d89f279b2b7f1"}
Sep 18 19:39:34.380374 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.380323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-52cq7" event={"ID":"8492f48b-11e8-4c35-88ea-d9eaf0f6de53","Type":"ContainerStarted","Data":"9b8ebec211a9da7913f817c43921ff5ab965a250f85064c9d7d812daae854473"}
Sep 18 19:39:34.381476 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.381456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pkqrg" event={"ID":"4fd05598-f32e-42b9-82a7-cb695c62d50c","Type":"ContainerStarted","Data":"e57a1c84811ceee51360a5983659dd4bf14629a2f014587a4637744f07f244e1"}
Sep 18 19:39:34.454272 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:34.454197    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dng7g"]
Sep 18 19:39:34.457868 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:34.457824    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e92495_9fea_41e7_8ec2_fdfb24825356.slice/crio-4a87a1386759c14bff0bd62381ed3716eb845aaa413f2315be73feb2abb6c606 WatchSource:0}: Error finding container 4a87a1386759c14bff0bd62381ed3716eb845aaa413f2315be73feb2abb6c606: Status 404 returned error can't find the container with id 4a87a1386759c14bff0bd62381ed3716eb845aaa413f2315be73feb2abb6c606
Sep 18 19:39:35.387890 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:35.387749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dng7g" event={"ID":"30e92495-9fea-41e7-8ec2-fdfb24825356","Type":"ContainerStarted","Data":"f1d7f5e86fcd3b389aeab4163daa6aa3de252313be35c3c397490b392acb9227"}
Sep 18 19:39:35.387890 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:35.387815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dng7g" event={"ID":"30e92495-9fea-41e7-8ec2-fdfb24825356","Type":"ContainerStarted","Data":"6e0ae674b3938991abdbf37e7b0325d751533bcc4b3a1271287ac4e5dc375cd6"}
Sep 18 19:39:35.387890 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:35.387848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dng7g" event={"ID":"30e92495-9fea-41e7-8ec2-fdfb24825356","Type":"ContainerStarted","Data":"4a87a1386759c14bff0bd62381ed3716eb845aaa413f2315be73feb2abb6c606"}
Sep 18 19:39:36.394463 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:36.394334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q65cx" event={"ID":"2f45aa15-f96d-4fc2-8c7b-4e96873d6147","Type":"ContainerStarted","Data":"13db4155c885e13b33c78f8957f8b54e32f7b7c0e87460730bd7dd5a66af8a5a"}
Sep 18 19:39:36.394463 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:36.394415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q65cx" event={"ID":"2f45aa15-f96d-4fc2-8c7b-4e96873d6147","Type":"ContainerStarted","Data":"dd6b571a389f9ad0afcb854bf67abd1335e9643e332225ab792c2f5e90d3b1ae"}
Sep 18 19:39:36.413828 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:36.413771    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q65cx" podStartSLOduration=67.204520468 podStartE2EDuration="1m8.413752909s" totalImagesPullingTime="1.209232441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:36.411951812 +0000 UTC m=+70.666158418" watchObservedRunningTime="2026-09-18 19:39:36.413752909 +0000 UTC m=+70.667959516"
Sep 18 19:39:39.406684 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:39.406651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-52cq7" event={"ID":"8492f48b-11e8-4c35-88ea-d9eaf0f6de53","Type":"ContainerStarted","Data":"98693d49725a1c5c97c5c16cd7786e999918eef90da5f2791fd57269e2244e1b"}
Sep 18 19:39:39.407102 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:39.406784    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:39:39.408305 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:39.408285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pkqrg" event={"ID":"4fd05598-f32e-42b9-82a7-cb695c62d50c","Type":"ContainerStarted","Data":"325ce362c6a923f1a78786fd1362309477593f75332374d1825f238618bb5b63"}
Sep 18 19:39:39.410470 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:39.410445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dng7g" event={"ID":"30e92495-9fea-41e7-8ec2-fdfb24825356","Type":"ContainerStarted","Data":"8152455e29b5d9d7805083e1c923c7aca9333f11a2c44b6f5b02de03a3142173"}
Sep 18 19:39:39.420570 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:39.420526    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-52cq7" podStartSLOduration=66.485011786 podStartE2EDuration="1m11.420513495s" totalImagesPullingTime="4.935501709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.420314936 +0000 UTC m=+73.674521540" watchObservedRunningTime="2026-09-18 19:39:39.420513495 +0000 UTC m=+73.674720099"
Sep 18 19:39:39.440371 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:39.437886    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dng7g" podStartSLOduration=33.032710054 podStartE2EDuration="37.437870262s" totalImagesPullingTime="4.405160208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.436507614 +0000 UTC m=+73.690714217" watchObservedRunningTime="2026-09-18 19:39:39.437870262 +0000 UTC m=+73.692076872"
Sep 18 19:39:39.456159 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:39.456125    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pkqrg" podStartSLOduration=66.504557595 podStartE2EDuration="1m11.456113463s" totalImagesPullingTime="4.951555868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.455628793 +0000 UTC m=+73.709835398" watchObservedRunningTime="2026-09-18 19:39:39.456113463 +0000 UTC m=+73.710320067"
Sep 18 19:39:41.829970 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.829936    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wjrfz"]
Sep 18 19:39:41.832157 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.832135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.834821 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.834799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:39:41.834970 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.834807    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:39:41.835064 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.834807    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qmpz8\""
Sep 18 19:39:41.835064 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.834807    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:39:41.835709 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.835692    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:39:41.835819 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.835730    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:39:41.835819 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.835759    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:39:41.866435 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-root\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866534 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-metrics-client-ca\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866534 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-tls\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866619 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-wtmp\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866619 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-sys\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866619 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-textfile\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866717 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866717 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbxd\" (UniqueName: \"kubernetes.io/projected/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-kube-api-access-lqbxd\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.866790 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.866728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967029 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-root\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967141 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-metrics-client-ca\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967141 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-tls\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967141 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-root\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967141 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-wtmp\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967141 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-sys\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967420 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-textfile\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967420 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967420 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbxd\" (UniqueName: \"kubernetes.io/projected/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-kube-api-access-lqbxd\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967420 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-wtmp\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967420 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-sys\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967420 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967707 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-textfile\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.967967 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.968046 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.967920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-metrics-client-ca\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.969610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.969583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.969738 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.969722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-node-exporter-tls\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:41.978424 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:41.978404    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbxd\" (UniqueName: \"kubernetes.io/projected/2f950c7c-633a-4ad7-83cf-2f2f50dc7e39-kube-api-access-lqbxd\") pod \"node-exporter-wjrfz\" (UID: \"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39\") " pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:42.141250 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.141172    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wjrfz"
Sep 18 19:39:42.152255 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:42.152232    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f950c7c_633a_4ad7_83cf_2f2f50dc7e39.slice/crio-1619090ea25e3a253fe30d676985982639d40aa2665f5ada396316f1e3334f67 WatchSource:0}: Error finding container 1619090ea25e3a253fe30d676985982639d40aa2665f5ada396316f1e3334f67: Status 404 returned error can't find the container with id 1619090ea25e3a253fe30d676985982639d40aa2665f5ada396316f1e3334f67
Sep 18 19:39:42.422610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.422530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjrfz" event={"ID":"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39","Type":"ContainerStarted","Data":"1619090ea25e3a253fe30d676985982639d40aa2665f5ada396316f1e3334f67"}
Sep 18 19:39:42.918008 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.917972    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:39:42.923205 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.922981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.927784 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.927762    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:39:42.927784 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.927781    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:39:42.927972 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.927873    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:39:42.927972 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.927881    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:39:42.927972 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.927917    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:39:42.928893 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.928876    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:39:42.928893 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.928884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k9cl7\""
Sep 18 19:39:42.929053 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.928933    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:39:42.929053 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.929029    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:39:42.929164 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.929060    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:39:42.941497 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.941457    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:39:42.974075 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974247 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974247 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-web-config\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974247 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974247 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmzc\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-kube-api-access-krmzc\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974449 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974449 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-config-out\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974449 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974586 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974750 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.974795 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.975005 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.974980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:42.975112 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:42.975019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076237 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-config-out\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076237 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076504 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076504 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076504 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076504 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076504 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076741 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:39:43.076568    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle podName:79b6e746-24a1-47aa-8c29-a10b83083594 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:43.57654049 +0000 UTC m=+77.830747071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594") : configmap references non-existent config key: ca-bundle.crt
Sep 18 19:39:43.076877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-web-config\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.076877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.077087 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmzc\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-kube-api-access-krmzc\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.077087 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.076936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.078498 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.077480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.078498 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.078118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.079508 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.079445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-config-out\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.080782 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.080594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.080883 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.080794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.081392 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.081369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.081703 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.081666    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-web-config\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.081919 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.081898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.081998 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.081979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.082163 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.082141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.082499 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.082479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-config-volume\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.087189 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.087167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmzc\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-kube-api-access-krmzc\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.426944 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.426913    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f950c7c-633a-4ad7-83cf-2f2f50dc7e39" containerID="bcd0d3e981226028d2fbfe107eab35826dd03fdabf5dab2cad0bffe7fb5521c7" exitCode=0
Sep 18 19:39:43.427054 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.426969    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjrfz" event={"ID":"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39","Type":"ContainerDied","Data":"bcd0d3e981226028d2fbfe107eab35826dd03fdabf5dab2cad0bffe7fb5521c7"}
Sep 18 19:39:43.581344 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.581310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.581989 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.581969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:43.835276 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:43.835245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:44.118030 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:44.117956    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:39:44.120893 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:44.120863    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6e746_24a1_47aa_8c29_a10b83083594.slice/crio-25933d0a08c199d3c71d11232ddcdf83ff731ff14a03a9160321135b43b56812 WatchSource:0}: Error finding container 25933d0a08c199d3c71d11232ddcdf83ff731ff14a03a9160321135b43b56812: Status 404 returned error can't find the container with id 25933d0a08c199d3c71d11232ddcdf83ff731ff14a03a9160321135b43b56812
Sep 18 19:39:44.431992 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:44.431913    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjrfz" event={"ID":"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39","Type":"ContainerStarted","Data":"247269eecb445e06f576ae86ebd281a245837de251d25689e8b1818b1a811901"}
Sep 18 19:39:44.431992 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:44.431960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjrfz" event={"ID":"2f950c7c-633a-4ad7-83cf-2f2f50dc7e39","Type":"ContainerStarted","Data":"a7851e8485fda47903b980bcacb6123e8717942f40f38291c728971282274a43"}
Sep 18 19:39:44.433212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:44.433188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerStarted","Data":"25933d0a08c199d3c71d11232ddcdf83ff731ff14a03a9160321135b43b56812"}
Sep 18 19:39:44.453171 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:44.453103    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wjrfz" podStartSLOduration=2.354125233 podStartE2EDuration="3.453072243s" totalImagesPullingTime="1.031355757s" totalInitContainerRuntime="67.591253ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:44.451901399 +0000 UTC m=+78.706108020" watchObservedRunningTime="2026-09-18 19:39:44.453072243 +0000 UTC m=+78.707278843"
Sep 18 19:39:45.438262 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:45.438231    2583 generic.go:356] "Generic (PLEG): container finished" podID="79b6e746-24a1-47aa-8c29-a10b83083594" containerID="ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d" exitCode=0
Sep 18 19:39:45.438726 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:45.438323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d"}
Sep 18 19:39:47.449231 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:47.449200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerStarted","Data":"e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0"}
Sep 18 19:39:47.449574 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:47.449294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerStarted","Data":"4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e"}
Sep 18 19:39:47.449574 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:47.449323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerStarted","Data":"458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082"}
Sep 18 19:39:47.449574 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:47.449348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerStarted","Data":"767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f"}
Sep 18 19:39:47.449574 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:47.449396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerStarted","Data":"f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86"}
Sep 18 19:39:48.159084 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.159012    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:39:48.161331 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.161315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.164037 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.163990    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:39:48.164037 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.164023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:39:48.164219 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.164064    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:39:48.164288 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.164255    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:39:48.164341 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.164321    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c50a5hvgg84b8\""
Sep 18 19:39:48.164424 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.164384    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:39:48.165061 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165042    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:39:48.165212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:39:48.165212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165108    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:39:48.165212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:39:48.165212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165093    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:39:48.165212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165208    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:39:48.165528 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165322    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-twfb5\""
Sep 18 19:39:48.165528 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:39:48.165528 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165513    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:39:48.165528 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.165520    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:39:48.166881 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.166858    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:39:48.183171 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.183150    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:39:48.214737 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.214708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.214879 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.214744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.214879 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.214771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.214879 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.214853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.214999 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.214918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.214999 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.214948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.214999 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.214975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-config\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215093 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215093 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215093 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215093 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215229 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215229 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-web-config\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215229 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215229 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-config-out\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215229 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215421 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215421 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215421 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5k7\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-kube-api-access-sb5k7\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.215421 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.215306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315756 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315756 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-web-config\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-config-out\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.315961 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5k7\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-kube-api-access-sb5k7\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.315996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-config\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316295 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.316808 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.316432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.317378 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.317014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.317378 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.317060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.317378 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.317346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.317895 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.317868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.319832 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.319657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.319953 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.319926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.320272 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.320247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.320431 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.320376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.320431 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.320415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.321120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.320675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.321120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.321084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.321120 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.321108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-web-config\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.321309 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.321154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.321415 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.321390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-config-out\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.321578 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.321530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-config\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.322304 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.322282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.322558 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.322540    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.323152 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.323130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.327548 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.327526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5k7\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-kube-api-access-sb5k7\") pod \"prometheus-k8s-0\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.456075 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.456010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerStarted","Data":"f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1"}
Sep 18 19:39:48.471347 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.471324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:48.764635 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.764581    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.450485237 podStartE2EDuration="6.764565016s" totalImagesPullingTime="3.299709418s" totalInitContainerRuntime="14.370361ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:48.491186249 +0000 UTC m=+82.745392852" watchObservedRunningTime="2026-09-18 19:39:48.764565016 +0000 UTC m=+83.018771619"
Sep 18 19:39:48.765675 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:48.765650    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:39:48.768497 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:39:48.768472    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb2a7b4_c7a8_4155_a654_6b9984398378.slice/crio-1b12cf3f5b8bfeb5176cf8d1240dc9d3bac6d9e61f3daa709cf8b11899713edf WatchSource:0}: Error finding container 1b12cf3f5b8bfeb5176cf8d1240dc9d3bac6d9e61f3daa709cf8b11899713edf: Status 404 returned error can't find the container with id 1b12cf3f5b8bfeb5176cf8d1240dc9d3bac6d9e61f3daa709cf8b11899713edf
Sep 18 19:39:49.461085 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:49.461047    2583 generic.go:356] "Generic (PLEG): container finished" podID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a" exitCode=0
Sep 18 19:39:49.461690 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:49.461139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"}
Sep 18 19:39:49.461690 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:49.461220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerStarted","Data":"1b12cf3f5b8bfeb5176cf8d1240dc9d3bac6d9e61f3daa709cf8b11899713edf"}
Sep 18 19:39:52.478819 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:52.478774    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerStarted","Data":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"}
Sep 18 19:39:52.479286 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:52.478870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerStarted","Data":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"}
Sep 18 19:39:54.489598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:54.489508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerStarted","Data":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"}
Sep 18 19:39:54.489598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:54.489582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerStarted","Data":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"}
Sep 18 19:39:54.490084 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:54.489620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerStarted","Data":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"}
Sep 18 19:39:54.490084 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:54.489655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerStarted","Data":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"}
Sep 18 19:39:54.522501 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:54.522446    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.163782831 podStartE2EDuration="6.522423858s" totalImagesPullingTime="4.340677526s" totalInitContainerRuntime="17.963501ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:54.520943499 +0000 UTC m=+88.775150105" watchObservedRunningTime="2026-09-18 19:39:54.522423858 +0000 UTC m=+88.776630463"
Sep 18 19:39:58.472325 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:39:58.472289    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:10.415150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:40:10.415123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-52cq7"
Sep 18 19:40:48.471830 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:40:48.471783    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:48.491265 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:40:48.491235    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:48.700892 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:40:48.700865    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:02.153643 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.153611    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:41:02.154014 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.153902    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="alertmanager" containerID="cri-o://f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86" gracePeriod=120
Sep 18 19:41:02.154014 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.153985    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-web" containerID="cri-o://458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082" gracePeriod=120
Sep 18 19:41:02.154135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.154022    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy" containerID="cri-o://4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e" gracePeriod=120
Sep 18 19:41:02.154135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.154018    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="config-reloader" containerID="cri-o://767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f" gracePeriod=120
Sep 18 19:41:02.154135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.154041    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="prom-label-proxy" containerID="cri-o://f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1" gracePeriod=120
Sep 18 19:41:02.154135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.153969    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-metric" containerID="cri-o://e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0" gracePeriod=120
Sep 18 19:41:02.738524 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738489    2583 generic.go:356] "Generic (PLEG): container finished" podID="79b6e746-24a1-47aa-8c29-a10b83083594" containerID="f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1" exitCode=0
Sep 18 19:41:02.738524 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738513    2583 generic.go:356] "Generic (PLEG): container finished" podID="79b6e746-24a1-47aa-8c29-a10b83083594" containerID="e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0" exitCode=0
Sep 18 19:41:02.738524 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738520    2583 generic.go:356] "Generic (PLEG): container finished" podID="79b6e746-24a1-47aa-8c29-a10b83083594" containerID="4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e" exitCode=0
Sep 18 19:41:02.738524 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738525    2583 generic.go:356] "Generic (PLEG): container finished" podID="79b6e746-24a1-47aa-8c29-a10b83083594" containerID="767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f" exitCode=0
Sep 18 19:41:02.738524 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738531    2583 generic.go:356] "Generic (PLEG): container finished" podID="79b6e746-24a1-47aa-8c29-a10b83083594" containerID="f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86" exitCode=0
Sep 18 19:41:02.738803 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1"}
Sep 18 19:41:02.738803 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0"}
Sep 18 19:41:02.738803 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e"}
Sep 18 19:41:02.738803 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f"}
Sep 18 19:41:02.738803 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:02.738703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86"}
Sep 18 19:41:03.393345 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.393324    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.462726 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462693    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-config-volume\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.462886 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462740    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krmzc\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-kube-api-access-krmzc\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.462886 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.462886 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462792    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-cluster-tls-config\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.462886 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462812    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-metric\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.462886 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462838    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-tls-assets\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.462886 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-web-config\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.463209 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462904    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.463209 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462934    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-metrics-client-ca\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.463209 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462960    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-config-out\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.463209 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.462997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-web\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.463209 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.463026    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-main-tls\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.463209 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.463078    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-main-db\") pod \"79b6e746-24a1-47aa-8c29-a10b83083594\" (UID: \"79b6e746-24a1-47aa-8c29-a10b83083594\") "
Sep 18 19:41:03.463528 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.463204    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:03.463528 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.463320    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.463738 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.463713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:03.464570 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.464510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:03.465751 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.465725    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-kube-api-access-krmzc" (OuterVolumeSpecName: "kube-api-access-krmzc") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "kube-api-access-krmzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:03.466746 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.466634    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:03.466746 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.466706    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:03.466941 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.466913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:03.467197 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.467171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-config-out" (OuterVolumeSpecName: "config-out") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:03.468068 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.468007    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-config-volume" (OuterVolumeSpecName: "config-volume") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:03.468552 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.468522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:03.468638 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.468584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:03.471875 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.471849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:03.478675 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.478604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-web-config" (OuterVolumeSpecName: "web-config") pod "79b6e746-24a1-47aa-8c29-a10b83083594" (UID: "79b6e746-24a1-47aa-8c29-a10b83083594"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:03.563973 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.563948    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-config-volume\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.563973 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.563971    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmzc\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-kube-api-access-krmzc\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.563981    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-cluster-tls-config\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.563991    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.563999    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79b6e746-24a1-47aa-8c29-a10b83083594-tls-assets\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.564007    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-web-config\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.564015    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.564024    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e746-24a1-47aa-8c29-a10b83083594-metrics-client-ca\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.564031    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-config-out\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.564039    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.564047    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79b6e746-24a1-47aa-8c29-a10b83083594-secret-alertmanager-main-tls\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.564105 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.564055    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79b6e746-24a1-47aa-8c29-a10b83083594-alertmanager-main-db\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:03.745231 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.745161    2583 generic.go:356] "Generic (PLEG): container finished" podID="79b6e746-24a1-47aa-8c29-a10b83083594" containerID="458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082" exitCode=0
Sep 18 19:41:03.745231 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.745206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082"}
Sep 18 19:41:03.745409 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.745241    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.745409 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.745261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79b6e746-24a1-47aa-8c29-a10b83083594","Type":"ContainerDied","Data":"25933d0a08c199d3c71d11232ddcdf83ff731ff14a03a9160321135b43b56812"}
Sep 18 19:41:03.745409 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.745295    2583 scope.go:117] "RemoveContainer" containerID="f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1"
Sep 18 19:41:03.756431 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.756381    2583 scope.go:117] "RemoveContainer" containerID="e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0"
Sep 18 19:41:03.763875 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.763856    2583 scope.go:117] "RemoveContainer" containerID="4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e"
Sep 18 19:41:03.771071 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.771034    2583 scope.go:117] "RemoveContainer" containerID="458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082"
Sep 18 19:41:03.772324 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.772303    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:41:03.775978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.775956    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:41:03.778507 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.778490    2583 scope.go:117] "RemoveContainer" containerID="767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f"
Sep 18 19:41:03.785113 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.785097    2583 scope.go:117] "RemoveContainer" containerID="f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86"
Sep 18 19:41:03.792074 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.792059    2583 scope.go:117] "RemoveContainer" containerID="ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d"
Sep 18 19:41:03.798411 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.798394    2583 scope.go:117] "RemoveContainer" containerID="f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1"
Sep 18 19:41:03.798677 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:03.798658    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1\": container with ID starting with f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1 not found: ID does not exist" containerID="f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1"
Sep 18 19:41:03.798722 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.798685    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1"} err="failed to get container status \"f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1\": rpc error: code = NotFound desc = could not find container \"f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1\": container with ID starting with f7bfd99dd20ed7c56696644915d5b5a29433c88514df9d7c965cc2a21ce3f5a1 not found: ID does not exist"
Sep 18 19:41:03.798722 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.798714    2583 scope.go:117] "RemoveContainer" containerID="e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0"
Sep 18 19:41:03.798943 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:03.798926    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0\": container with ID starting with e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0 not found: ID does not exist" containerID="e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0"
Sep 18 19:41:03.799000 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.798950    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0"} err="failed to get container status \"e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0\": rpc error: code = NotFound desc = could not find container \"e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0\": container with ID starting with e773456548b9079b951d014d7053b5e29171a7bd0774a64f56c5a74e4d56e5a0 not found: ID does not exist"
Sep 18 19:41:03.799000 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.798965    2583 scope.go:117] "RemoveContainer" containerID="4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e"
Sep 18 19:41:03.799212 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:03.799195    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e\": container with ID starting with 4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e not found: ID does not exist" containerID="4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e"
Sep 18 19:41:03.799259 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.799218    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e"} err="failed to get container status \"4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e\": rpc error: code = NotFound desc = could not find container \"4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e\": container with ID starting with 4e59825c19f6f665612a7cc77040da33a00f7e1ce0b4c92feb85c9bac208c56e not found: ID does not exist"
Sep 18 19:41:03.799259 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.799233    2583 scope.go:117] "RemoveContainer" containerID="458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082"
Sep 18 19:41:03.799469 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:03.799449    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082\": container with ID starting with 458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082 not found: ID does not exist" containerID="458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082"
Sep 18 19:41:03.799525 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.799478    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082"} err="failed to get container status \"458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082\": rpc error: code = NotFound desc = could not find container \"458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082\": container with ID starting with 458e95c8e97574980c6df5b69f39ed7a2bfbd9c5690c33ae58b107c09669e082 not found: ID does not exist"
Sep 18 19:41:03.799525 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.799492    2583 scope.go:117] "RemoveContainer" containerID="767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f"
Sep 18 19:41:03.799760 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:03.799743    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f\": container with ID starting with 767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f not found: ID does not exist" containerID="767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f"
Sep 18 19:41:03.799865 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.799765    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f"} err="failed to get container status \"767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f\": rpc error: code = NotFound desc = could not find container \"767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f\": container with ID starting with 767b70e9c37b1d6e5d552e1fc8c26da7d71cd5b7c859f380023ef2c82cade13f not found: ID does not exist"
Sep 18 19:41:03.799865 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.799783    2583 scope.go:117] "RemoveContainer" containerID="f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86"
Sep 18 19:41:03.800331 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:03.800109    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86\": container with ID starting with f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86 not found: ID does not exist" containerID="f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86"
Sep 18 19:41:03.800331 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.800159    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86"} err="failed to get container status \"f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86\": rpc error: code = NotFound desc = could not find container \"f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86\": container with ID starting with f9e02c62fe30bafc0378d15cf6a4bab4b6b3bf6897e6b3085add6d9c4cbb6a86 not found: ID does not exist"
Sep 18 19:41:03.800331 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.800177    2583 scope.go:117] "RemoveContainer" containerID="ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d"
Sep 18 19:41:03.800497 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:03.800460    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d\": container with ID starting with ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d not found: ID does not exist" containerID="ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d"
Sep 18 19:41:03.800497 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.800481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d"} err="failed to get container status \"ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d\": rpc error: code = NotFound desc = could not find container \"ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d\": container with ID starting with ca04a72b75e91a55a09e2ff75f70062de38a8c34dd16bf30bb74639b937dc45d not found: ID does not exist"
Sep 18 19:41:03.964958 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:03.964920    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" path="/var/lib/kubelet/pods/79b6e746-24a1-47aa-8c29-a10b83083594/volumes"
Sep 18 19:41:06.490185 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.490150    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:41:06.490595 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.490462    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="prometheus" containerID="cri-o://f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831" gracePeriod=600
Sep 18 19:41:06.490595 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.490492    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy" containerID="cri-o://b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7" gracePeriod=600
Sep 18 19:41:06.490595 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.490542    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-thanos" containerID="cri-o://71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9" gracePeriod=600
Sep 18 19:41:06.490747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.490571    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-web" containerID="cri-o://16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9" gracePeriod=600
Sep 18 19:41:06.490747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.490544    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="config-reloader" containerID="cri-o://011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d" gracePeriod=600
Sep 18 19:41:06.490747 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.490543    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="thanos-sidecar" containerID="cri-o://18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b" gracePeriod=600
Sep 18 19:41:06.726833 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.726812    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:06.759914 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759887    2583 generic.go:356] "Generic (PLEG): container finished" podID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9" exitCode=0
Sep 18 19:41:06.759914 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759907    2583 generic.go:356] "Generic (PLEG): container finished" podID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7" exitCode=0
Sep 18 19:41:06.759914 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759913    2583 generic.go:356] "Generic (PLEG): container finished" podID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9" exitCode=0
Sep 18 19:41:06.759914 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759919    2583 generic.go:356] "Generic (PLEG): container finished" podID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b" exitCode=0
Sep 18 19:41:06.760150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759926    2583 generic.go:356] "Generic (PLEG): container finished" podID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d" exitCode=0
Sep 18 19:41:06.760150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759931    2583 generic.go:356] "Generic (PLEG): container finished" podID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831" exitCode=0
Sep 18 19:41:06.760150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759977    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:06.760150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.759978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"}
Sep 18 19:41:06.760150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.760067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"}
Sep 18 19:41:06.760150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.760110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"}
Sep 18 19:41:06.760150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.760141    2583 scope.go:117] "RemoveContainer" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.760505 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.760143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"}
Sep 18 19:41:06.760505 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.760287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"}
Sep 18 19:41:06.760505 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.760339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"}
Sep 18 19:41:06.760505 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.760404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cdb2a7b4-c7a8-4155-a654-6b9984398378","Type":"ContainerDied","Data":"1b12cf3f5b8bfeb5176cf8d1240dc9d3bac6d9e61f3daa709cf8b11899713edf"}
Sep 18 19:41:06.768688 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.768664    2583 scope.go:117] "RemoveContainer" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.777491 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.777470    2583 scope.go:117] "RemoveContainer" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.784397 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.784341    2583 scope.go:117] "RemoveContainer" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.787552 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.787533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.787624 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.787559    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-tls\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.787624 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.787579    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-config-out\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.787624 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.787595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-1\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.787624 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.787617    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-thanos-prometheus-http-client-file\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788299 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788224    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:06.788299 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-2\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788299 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788350    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-serving-certs-ca-bundle\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788588 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788398    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-kube-rbac-proxy\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788588 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-grpc-tls\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788792 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:06.788852 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788823    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-trusted-ca-bundle\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788909 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788860    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb5k7\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-kube-api-access-sb5k7\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-kubelet-serving-ca-bundle\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.788963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.788936    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-metrics-client-certs\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789316 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789288    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:06.789578 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-web-config\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789578 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789401    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-config\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789578 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789503    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-0\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789578 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-tls-assets\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789578 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789570    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-metrics-client-ca\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789850 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789601    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789850 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-db\") pod \"cdb2a7b4-c7a8-4155-a654-6b9984398378\" (UID: \"cdb2a7b4-c7a8-4155-a654-6b9984398378\") "
Sep 18 19:41:06.789850 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789832    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.789850 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789847    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.790050 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789862    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.790822 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789539    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:06.790907 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.789827    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:06.790907 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.790789    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:06.791776 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.791715    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.792029 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.792005    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-config-out" (OuterVolumeSpecName: "config-out") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:06.792103 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.792066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:06.792777 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.792754    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:06.792913 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.792887    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.793304 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.793276    2583 scope.go:117] "RemoveContainer" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.794007 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.793972    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-kube-api-access-sb5k7" (OuterVolumeSpecName: "kube-api-access-sb5k7") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "kube-api-access-sb5k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:06.794499 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.794374    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-config" (OuterVolumeSpecName: "config") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.794600 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.794507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.794600 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.794590    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.794825 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.794683    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.794958 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.794937    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:06.795706 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.795672    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.796002 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.795981    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.802102 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.802078    2583 scope.go:117] "RemoveContainer" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.803216 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.803193    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-web-config" (OuterVolumeSpecName: "web-config") pod "cdb2a7b4-c7a8-4155-a654-6b9984398378" (UID: "cdb2a7b4-c7a8-4155-a654-6b9984398378"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:06.809881 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.809861    2583 scope.go:117] "RemoveContainer" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.817197 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.817181    2583 scope.go:117] "RemoveContainer" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.817500 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:06.817481    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": container with ID starting with 71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9 not found: ID does not exist" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.817552 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.817511    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"} err="failed to get container status \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": rpc error: code = NotFound desc = could not find container \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": container with ID starting with 71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9 not found: ID does not exist"
Sep 18 19:41:06.817552 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.817529    2583 scope.go:117] "RemoveContainer" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.817806 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:06.817792    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": container with ID starting with b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7 not found: ID does not exist" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.817865 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.817809    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"} err="failed to get container status \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": rpc error: code = NotFound desc = could not find container \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": container with ID starting with b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7 not found: ID does not exist"
Sep 18 19:41:06.817865 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.817820    2583 scope.go:117] "RemoveContainer" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.818064 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:06.818045    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": container with ID starting with 16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9 not found: ID does not exist" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.818106 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818070    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"} err="failed to get container status \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": rpc error: code = NotFound desc = could not find container \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": container with ID starting with 16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9 not found: ID does not exist"
Sep 18 19:41:06.818106 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818085    2583 scope.go:117] "RemoveContainer" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.818316 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:06.818300    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": container with ID starting with 18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b not found: ID does not exist" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.818411 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818333    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"} err="failed to get container status \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": rpc error: code = NotFound desc = could not find container \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": container with ID starting with 18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b not found: ID does not exist"
Sep 18 19:41:06.818411 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818370    2583 scope.go:117] "RemoveContainer" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.818660 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:06.818644    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": container with ID starting with 011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d not found: ID does not exist" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.818723 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818666    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"} err="failed to get container status \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": rpc error: code = NotFound desc = could not find container \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": container with ID starting with 011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d not found: ID does not exist"
Sep 18 19:41:06.818723 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818688    2583 scope.go:117] "RemoveContainer" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.818920 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:06.818902    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": container with ID starting with f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831 not found: ID does not exist" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.818996 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818926    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"} err="failed to get container status \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": rpc error: code = NotFound desc = could not find container \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": container with ID starting with f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831 not found: ID does not exist"
Sep 18 19:41:06.818996 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.818944    2583 scope.go:117] "RemoveContainer" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.819219 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:41:06.819199    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": container with ID starting with daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a not found: ID does not exist" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.819281 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819224    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"} err="failed to get container status \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": rpc error: code = NotFound desc = could not find container \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": container with ID starting with daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a not found: ID does not exist"
Sep 18 19:41:06.819281 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819241    2583 scope.go:117] "RemoveContainer" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.819471 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819454    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"} err="failed to get container status \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": rpc error: code = NotFound desc = could not find container \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": container with ID starting with 71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9 not found: ID does not exist"
Sep 18 19:41:06.819541 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819472    2583 scope.go:117] "RemoveContainer" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.819717 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819701    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"} err="failed to get container status \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": rpc error: code = NotFound desc = could not find container \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": container with ID starting with b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7 not found: ID does not exist"
Sep 18 19:41:06.819782 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819717    2583 scope.go:117] "RemoveContainer" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.819919 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819901    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"} err="failed to get container status \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": rpc error: code = NotFound desc = could not find container \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": container with ID starting with 16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9 not found: ID does not exist"
Sep 18 19:41:06.819977 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.819920    2583 scope.go:117] "RemoveContainer" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.820138 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820122    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"} err="failed to get container status \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": rpc error: code = NotFound desc = could not find container \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": container with ID starting with 18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b not found: ID does not exist"
Sep 18 19:41:06.820198 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820139    2583 scope.go:117] "RemoveContainer" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.820345 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820322    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"} err="failed to get container status \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": rpc error: code = NotFound desc = could not find container \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": container with ID starting with 011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d not found: ID does not exist"
Sep 18 19:41:06.820437 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820346    2583 scope.go:117] "RemoveContainer" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.820615 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820597    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"} err="failed to get container status \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": rpc error: code = NotFound desc = could not find container \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": container with ID starting with f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831 not found: ID does not exist"
Sep 18 19:41:06.820666 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820625    2583 scope.go:117] "RemoveContainer" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.820877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820859    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"} err="failed to get container status \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": rpc error: code = NotFound desc = could not find container \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": container with ID starting with daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a not found: ID does not exist"
Sep 18 19:41:06.820930 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.820878    2583 scope.go:117] "RemoveContainer" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.821111 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821095    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"} err="failed to get container status \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": rpc error: code = NotFound desc = could not find container \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": container with ID starting with 71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9 not found: ID does not exist"
Sep 18 19:41:06.821164 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821112    2583 scope.go:117] "RemoveContainer" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.821327 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821311    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"} err="failed to get container status \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": rpc error: code = NotFound desc = could not find container \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": container with ID starting with b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7 not found: ID does not exist"
Sep 18 19:41:06.821464 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821328    2583 scope.go:117] "RemoveContainer" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.821641 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821623    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"} err="failed to get container status \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": rpc error: code = NotFound desc = could not find container \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": container with ID starting with 16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9 not found: ID does not exist"
Sep 18 19:41:06.821718 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821656    2583 scope.go:117] "RemoveContainer" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.821908 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821892    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"} err="failed to get container status \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": rpc error: code = NotFound desc = could not find container \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": container with ID starting with 18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b not found: ID does not exist"
Sep 18 19:41:06.821978 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.821910    2583 scope.go:117] "RemoveContainer" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.822143 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822126    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"} err="failed to get container status \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": rpc error: code = NotFound desc = could not find container \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": container with ID starting with 011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d not found: ID does not exist"
Sep 18 19:41:06.822208 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822144    2583 scope.go:117] "RemoveContainer" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.822375 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822340    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"} err="failed to get container status \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": rpc error: code = NotFound desc = could not find container \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": container with ID starting with f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831 not found: ID does not exist"
Sep 18 19:41:06.822375 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822372    2583 scope.go:117] "RemoveContainer" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.822601 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822585    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"} err="failed to get container status \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": rpc error: code = NotFound desc = could not find container \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": container with ID starting with daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a not found: ID does not exist"
Sep 18 19:41:06.822669 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822602    2583 scope.go:117] "RemoveContainer" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.822810 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822793    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"} err="failed to get container status \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": rpc error: code = NotFound desc = could not find container \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": container with ID starting with 71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9 not found: ID does not exist"
Sep 18 19:41:06.822875 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.822811    2583 scope.go:117] "RemoveContainer" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.823071 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823051    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"} err="failed to get container status \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": rpc error: code = NotFound desc = could not find container \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": container with ID starting with b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7 not found: ID does not exist"
Sep 18 19:41:06.823117 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823071    2583 scope.go:117] "RemoveContainer" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.823304 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823288    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"} err="failed to get container status \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": rpc error: code = NotFound desc = could not find container \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": container with ID starting with 16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9 not found: ID does not exist"
Sep 18 19:41:06.823386 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823305    2583 scope.go:117] "RemoveContainer" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.823590 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823573    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"} err="failed to get container status \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": rpc error: code = NotFound desc = could not find container \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": container with ID starting with 18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b not found: ID does not exist"
Sep 18 19:41:06.823633 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823591    2583 scope.go:117] "RemoveContainer" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.823797 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823781    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"} err="failed to get container status \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": rpc error: code = NotFound desc = could not find container \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": container with ID starting with 011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d not found: ID does not exist"
Sep 18 19:41:06.823854 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.823798    2583 scope.go:117] "RemoveContainer" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.824049 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.824026    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"} err="failed to get container status \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": rpc error: code = NotFound desc = could not find container \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": container with ID starting with f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831 not found: ID does not exist"
Sep 18 19:41:06.824095 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.824053    2583 scope.go:117] "RemoveContainer" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.825529 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.825498    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"} err="failed to get container status \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": rpc error: code = NotFound desc = could not find container \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": container with ID starting with daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a not found: ID does not exist"
Sep 18 19:41:06.825529 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.825527    2583 scope.go:117] "RemoveContainer" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.827243 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.827219    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"} err="failed to get container status \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": rpc error: code = NotFound desc = could not find container \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": container with ID starting with 71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9 not found: ID does not exist"
Sep 18 19:41:06.827330 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.827246    2583 scope.go:117] "RemoveContainer" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.827557 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.827535    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"} err="failed to get container status \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": rpc error: code = NotFound desc = could not find container \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": container with ID starting with b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7 not found: ID does not exist"
Sep 18 19:41:06.827557 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.827557    2583 scope.go:117] "RemoveContainer" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.827843 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.827816    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"} err="failed to get container status \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": rpc error: code = NotFound desc = could not find container \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": container with ID starting with 16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9 not found: ID does not exist"
Sep 18 19:41:06.827843 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.827841    2583 scope.go:117] "RemoveContainer" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.828114 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828093    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"} err="failed to get container status \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": rpc error: code = NotFound desc = could not find container \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": container with ID starting with 18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b not found: ID does not exist"
Sep 18 19:41:06.828114 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828113    2583 scope.go:117] "RemoveContainer" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.828406 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828383    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"} err="failed to get container status \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": rpc error: code = NotFound desc = could not find container \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": container with ID starting with 011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d not found: ID does not exist"
Sep 18 19:41:06.828406 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828405    2583 scope.go:117] "RemoveContainer" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.828675 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828657    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"} err="failed to get container status \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": rpc error: code = NotFound desc = could not find container \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": container with ID starting with f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831 not found: ID does not exist"
Sep 18 19:41:06.828716 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828675    2583 scope.go:117] "RemoveContainer" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.828897 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828881    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"} err="failed to get container status \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": rpc error: code = NotFound desc = could not find container \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": container with ID starting with daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a not found: ID does not exist"
Sep 18 19:41:06.828897 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.828897    2583 scope.go:117] "RemoveContainer" containerID="71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"
Sep 18 19:41:06.829119 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829104    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9"} err="failed to get container status \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": rpc error: code = NotFound desc = could not find container \"71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9\": container with ID starting with 71dcf7a31e825a460b0a2c84778ad723b2894558ea596a6a5c033cf94d7231a9 not found: ID does not exist"
Sep 18 19:41:06.829186 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829119    2583 scope.go:117] "RemoveContainer" containerID="b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"
Sep 18 19:41:06.829327 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829304    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7"} err="failed to get container status \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": rpc error: code = NotFound desc = could not find container \"b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7\": container with ID starting with b8e7160cdf999d875cdc0e2851c2ec7e7961c4d9d77dc5af84d5d1121d9b0ac7 not found: ID does not exist"
Sep 18 19:41:06.829384 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829331    2583 scope.go:117] "RemoveContainer" containerID="16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"
Sep 18 19:41:06.829587 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829572    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9"} err="failed to get container status \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": rpc error: code = NotFound desc = could not find container \"16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9\": container with ID starting with 16593b22cbd9f8c6decee3f83704820be49f3af7fdd154c790ebe7e4db28a2b9 not found: ID does not exist"
Sep 18 19:41:06.829644 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829587    2583 scope.go:117] "RemoveContainer" containerID="18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"
Sep 18 19:41:06.829832 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b"} err="failed to get container status \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": rpc error: code = NotFound desc = could not find container \"18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b\": container with ID starting with 18d2a620a7676ffcd90fab5319eb0046357994b38556e02b6e9a29705211f42b not found: ID does not exist"
Sep 18 19:41:06.829884 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.829833    2583 scope.go:117] "RemoveContainer" containerID="011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"
Sep 18 19:41:06.830073 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.830058    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d"} err="failed to get container status \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": rpc error: code = NotFound desc = could not find container \"011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d\": container with ID starting with 011a1a14e6bc1c3ba9217bc2eb27b001c0e94d655814ab3b03aaf2e7a4be545d not found: ID does not exist"
Sep 18 19:41:06.830073 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.830074    2583 scope.go:117] "RemoveContainer" containerID="f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"
Sep 18 19:41:06.830271 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.830255    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831"} err="failed to get container status \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": rpc error: code = NotFound desc = could not find container \"f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831\": container with ID starting with f9963f546dbc32857fce4023ebd0a2dd1b4fc1ffcd6ce0aae23eb10dd5fa0831 not found: ID does not exist"
Sep 18 19:41:06.830309 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.830271    2583 scope.go:117] "RemoveContainer" containerID="daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"
Sep 18 19:41:06.830549 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.830534    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a"} err="failed to get container status \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": rpc error: code = NotFound desc = could not find container \"daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a\": container with ID starting with daeb1f3d6775535b18b28fae61a84ba09c86b5e4ea65ddb5dd696239dfd15f9a not found: ID does not exist"
Sep 18 19:41:06.890370 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890338    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-config\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890374    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890385    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-tls-assets\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890394    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-metrics-client-ca\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890404    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890413    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-db\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890422    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890433 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890432    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890439    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdb2a7b4-c7a8-4155-a654-6b9984398378-config-out\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890447    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890455    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890463    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-kube-rbac-proxy\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890471    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-grpc-tls\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890479    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5k7\" (UniqueName: \"kubernetes.io/projected/cdb2a7b4-c7a8-4155-a654-6b9984398378-kube-api-access-sb5k7\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890486    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb2a7b4-c7a8-4155-a654-6b9984398378-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890495    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-secret-metrics-client-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:06.890654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:06.890502    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdb2a7b4-c7a8-4155-a654-6b9984398378-web-config\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:41:07.123619 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.123592    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:41:07.134363 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.134324    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:41:07.173541 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173517    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:41:07.173753 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173742    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-thanos"
Sep 18 19:41:07.173794 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173755    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-thanos"
Sep 18 19:41:07.173794 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173764    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="alertmanager"
Sep 18 19:41:07.173794 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173769    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="alertmanager"
Sep 18 19:41:07.173794 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173776    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="prometheus"
Sep 18 19:41:07.173794 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173781    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="prometheus"
Sep 18 19:41:07.173794 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173789    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="init-config-reloader"
Sep 18 19:41:07.173794 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173794    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="init-config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173800    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-metric"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173805    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-metric"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173815    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173820    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173827    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-web"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173832    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-web"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173837    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173841    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173849    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173854    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173860    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-web"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173864    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-web"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173869    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="prom-label-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173874    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="prom-label-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173881    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="init-config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173886    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="init-config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173892    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173896    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173901    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="thanos-sidecar"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173905    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="thanos-sidecar"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173942    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="prom-label-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173948    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-web"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173953    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-web"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173958    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173963    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173969    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="thanos-sidecar"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173974    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="kube-rbac-proxy-metric"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173979    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="prometheus"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173983    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy-thanos"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173988    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="alertmanager"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173993    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6e746-24a1-47aa-8c29-a10b83083594" containerName="config-reloader"
Sep 18 19:41:07.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.173997    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" containerName="kube-rbac-proxy"
Sep 18 19:41:07.179141 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.179124    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.181720 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.181703    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:41:07.181834 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.181789    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:41:07.182055 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182029    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:41:07.182055 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c50a5hvgg84b8\""
Sep 18 19:41:07.182179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182078    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:41:07.182179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182057    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:41:07.182179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182029    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:41:07.182405 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182180    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:41:07.182405 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182184    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-twfb5\""
Sep 18 19:41:07.182405 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182260    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:41:07.182405 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182284    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:41:07.182405 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:41:07.182405 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182332    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:41:07.182756 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182668    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:41:07.182888 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.182862    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:41:07.185499 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.185480    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:41:07.187596 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.187580    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:41:07.192700 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192765 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192765 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2cf24ca2-cc03-414b-8506-6657d00c7b3a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192765 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192857 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192857 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192857 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-config\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192857 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2cf24ca2-cc03-414b-8506-6657d00c7b3a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wq4\" (UniqueName: \"kubernetes.io/projected/2cf24ca2-cc03-414b-8506-6657d00c7b3a-kube-api-access-85wq4\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.192963 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.193160 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.192983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.193160 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.193012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.193160 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.193025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.193160 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.193039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.193160 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.193061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.196622 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.196597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:41:07.294244 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294383 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294383 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2cf24ca2-cc03-414b-8506-6657d00c7b3a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294383 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294383 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-config\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2cf24ca2-cc03-414b-8506-6657d00c7b3a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294598 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wq4\" (UniqueName: \"kubernetes.io/projected/2cf24ca2-cc03-414b-8506-6657d00c7b3a-kube-api-access-85wq4\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.294876 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.295302 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.295207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.295404    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.294652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.296027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.296233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.296704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.296935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297666 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.297629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.297754 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.297734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.298084 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.298066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.298172 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.298085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.299210 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.299167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2cf24ca2-cc03-414b-8506-6657d00c7b3a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.299307 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.299212    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2cf24ca2-cc03-414b-8506-6657d00c7b3a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.299549 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.299528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2cf24ca2-cc03-414b-8506-6657d00c7b3a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.299636 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.299583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.299892 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.299872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.300735 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.300719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.300791 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.300770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.300854 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.300839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2cf24ca2-cc03-414b-8506-6657d00c7b3a-config\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.304723 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.304703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wq4\" (UniqueName: \"kubernetes.io/projected/2cf24ca2-cc03-414b-8506-6657d00c7b3a-kube-api-access-85wq4\") pod \"prometheus-k8s-0\" (UID: \"2cf24ca2-cc03-414b-8506-6657d00c7b3a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.489452 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.489349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:07.768709 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.768687    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:41:07.770805 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:41:07.770780    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf24ca2_cc03_414b_8506_6657d00c7b3a.slice/crio-8a551f40fe4c2d519a82a499c65c6f5d701949f11e9250181d3a372593225197 WatchSource:0}: Error finding container 8a551f40fe4c2d519a82a499c65c6f5d701949f11e9250181d3a372593225197: Status 404 returned error can't find the container with id 8a551f40fe4c2d519a82a499c65c6f5d701949f11e9250181d3a372593225197
Sep 18 19:41:07.966076 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:07.966048    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb2a7b4-c7a8-4155-a654-6b9984398378" path="/var/lib/kubelet/pods/cdb2a7b4-c7a8-4155-a654-6b9984398378/volumes"
Sep 18 19:41:08.769982 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:08.769948    2583 generic.go:356] "Generic (PLEG): container finished" podID="2cf24ca2-cc03-414b-8506-6657d00c7b3a" containerID="1c40f209e949225c8b3a8951b352cfd716375dfab54dcddefc083bf26e8952d9" exitCode=0
Sep 18 19:41:08.770322 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:08.770005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerDied","Data":"1c40f209e949225c8b3a8951b352cfd716375dfab54dcddefc083bf26e8952d9"}
Sep 18 19:41:08.770322 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:08.770056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerStarted","Data":"8a551f40fe4c2d519a82a499c65c6f5d701949f11e9250181d3a372593225197"}
Sep 18 19:41:09.778425 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:09.778389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerStarted","Data":"4bb09e6995f554c1c7957dd73de2161d203df3087de71ab5b9475aa8ad3a9b7a"}
Sep 18 19:41:09.778809 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:09.778480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerStarted","Data":"5bf97a9dc3caca9373fac6bb965d1b085f2e3ffc3d50a990c9bb212714c73e36"}
Sep 18 19:41:09.778809 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:09.778506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerStarted","Data":"f1c60843783cbdc76ee09669cf53e9c1270cd47845aa8c0218640ec1c2d875fd"}
Sep 18 19:41:09.778809 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:09.778538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerStarted","Data":"2d8e459dc956561ca3b8bcd24cc3bdea1cbfa4337c37f9d7476b1d4d0c523b67"}
Sep 18 19:41:09.778809 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:09.778563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerStarted","Data":"71a55ac42e4b946a3a419e4a2cdb20b0470bc26e767952d7d12a12870679381d"}
Sep 18 19:41:09.778809 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:09.778596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2cf24ca2-cc03-414b-8506-6657d00c7b3a","Type":"ContainerStarted","Data":"9df5cac4e2d07404a04a54683ec99d003ce2d3db4d6a723c7eedfadefeb47a80"}
Sep 18 19:41:09.805682 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:09.805634    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.787349772 podStartE2EDuration="2.805606496s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.256724ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:09.803847909 +0000 UTC m=+164.058054513" watchObservedRunningTime="2026-09-18 19:41:09.805606496 +0000 UTC m=+164.059813100"
Sep 18 19:41:12.490313 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:41:12.490276    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:42:07.490082 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:42:07.490044    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:42:07.505775 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:42:07.505753    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:42:07.987950 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:42:07.987925    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:27.845532 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:43:27.845501    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:43:27.845532 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:43:27.845528    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:43:27.851728 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:43:27.851707    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:45:15.622734 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.622704    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9rtmm"]
Sep 18 19:45:15.625644 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.625628    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:15.628199 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.628178    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 19:45:15.628318 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.628231    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 19:45:15.629094 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.629075    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gfrg2\""
Sep 18 19:45:15.634809 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.634787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9rtmm"]
Sep 18 19:45:15.722145 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.722121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39bb5a6d-ee47-4ce7-b07f-1de0edab31f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-9rtmm\" (UID: \"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6\") " pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:15.722237 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.722157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hh25\" (UniqueName: \"kubernetes.io/projected/39bb5a6d-ee47-4ce7-b07f-1de0edab31f6-kube-api-access-6hh25\") pod \"cert-manager-55b4d954d5-9rtmm\" (UID: \"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6\") " pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:15.822533 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.822512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39bb5a6d-ee47-4ce7-b07f-1de0edab31f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-9rtmm\" (UID: \"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6\") " pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:15.822617 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.822546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hh25\" (UniqueName: \"kubernetes.io/projected/39bb5a6d-ee47-4ce7-b07f-1de0edab31f6-kube-api-access-6hh25\") pod \"cert-manager-55b4d954d5-9rtmm\" (UID: \"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6\") " pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:15.830158 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.830130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39bb5a6d-ee47-4ce7-b07f-1de0edab31f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-9rtmm\" (UID: \"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6\") " pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:15.830419 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.830399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hh25\" (UniqueName: \"kubernetes.io/projected/39bb5a6d-ee47-4ce7-b07f-1de0edab31f6-kube-api-access-6hh25\") pod \"cert-manager-55b4d954d5-9rtmm\" (UID: \"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6\") " pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:15.934485 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:15.934432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9rtmm"
Sep 18 19:45:16.200812 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:16.200788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9rtmm"]
Sep 18 19:45:16.203185 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:45:16.203160    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39bb5a6d_ee47_4ce7_b07f_1de0edab31f6.slice/crio-fd9c3302ef691310596d4edb45d1f531a38948a78a1938e5dea02e2be83104da WatchSource:0}: Error finding container fd9c3302ef691310596d4edb45d1f531a38948a78a1938e5dea02e2be83104da: Status 404 returned error can't find the container with id fd9c3302ef691310596d4edb45d1f531a38948a78a1938e5dea02e2be83104da
Sep 18 19:45:16.205089 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:16.205073    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:45:16.603305 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:16.603275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9rtmm" event={"ID":"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6","Type":"ContainerStarted","Data":"fd9c3302ef691310596d4edb45d1f531a38948a78a1938e5dea02e2be83104da"}
Sep 18 19:45:19.615576 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:19.615540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9rtmm" event={"ID":"39bb5a6d-ee47-4ce7-b07f-1de0edab31f6","Type":"ContainerStarted","Data":"8bf2d3e1f8c0ff7797e193699b3f0fff7d7bd65bde36c5c57d40425fc1627d96"}
Sep 18 19:45:19.631515 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:19.631459    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9rtmm" podStartSLOduration=1.628817194 podStartE2EDuration="4.631445562s" totalImagesPullingTime="3.002628368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:19.631253315 +0000 UTC m=+413.885459934" watchObservedRunningTime="2026-09-18 19:45:19.631445562 +0000 UTC m=+413.885652165"
Sep 18 19:45:43.602875 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.602801    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"]
Sep 18 19:45:43.605797 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.605782    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.608612 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.608592    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:45:43.608746 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.608715    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 19:45:43.609692 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.609674    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 19:45:43.609810 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.609714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9xlpb\""
Sep 18 19:45:43.609810 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.609738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 19:45:43.610037 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.610022    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 19:45:43.616397 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.616377    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"]
Sep 18 19:45:43.710327 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.710293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8cab9b4-b072-42f9-9c57-1986f36fd954-metrics-cert\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.710490 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.710332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b8cab9b4-b072-42f9-9c57-1986f36fd954-manager-config\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.710490 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.710434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwj4t\" (UniqueName: \"kubernetes.io/projected/b8cab9b4-b072-42f9-9c57-1986f36fd954-kube-api-access-cwj4t\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.710490 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.710460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8cab9b4-b072-42f9-9c57-1986f36fd954-cert\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.810779 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.810744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b8cab9b4-b072-42f9-9c57-1986f36fd954-manager-config\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.810976 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.810828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwj4t\" (UniqueName: \"kubernetes.io/projected/b8cab9b4-b072-42f9-9c57-1986f36fd954-kube-api-access-cwj4t\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.810976 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.810860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8cab9b4-b072-42f9-9c57-1986f36fd954-cert\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.810976 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.810883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8cab9b4-b072-42f9-9c57-1986f36fd954-metrics-cert\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.811516 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.811495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b8cab9b4-b072-42f9-9c57-1986f36fd954-manager-config\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.813440 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.813418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8cab9b4-b072-42f9-9c57-1986f36fd954-cert\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.813595 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.813576    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8cab9b4-b072-42f9-9c57-1986f36fd954-metrics-cert\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.825092 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.825073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwj4t\" (UniqueName: \"kubernetes.io/projected/b8cab9b4-b072-42f9-9c57-1986f36fd954-kube-api-access-cwj4t\") pod \"lws-controller-manager-d69b774b8-9pb2m\" (UID: \"b8cab9b4-b072-42f9-9c57-1986f36fd954\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:43.915260 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:43.915192    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:44.187914 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:44.187891    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"]
Sep 18 19:45:44.190296 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:45:44.190259    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cab9b4_b072_42f9_9c57_1986f36fd954.slice/crio-d79eb3fc2bd95bc1a0e0eedeb271737b6066aa450763e2716d56be37380b453c WatchSource:0}: Error finding container d79eb3fc2bd95bc1a0e0eedeb271737b6066aa450763e2716d56be37380b453c: Status 404 returned error can't find the container with id d79eb3fc2bd95bc1a0e0eedeb271737b6066aa450763e2716d56be37380b453c
Sep 18 19:45:44.701266 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:44.701233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m" event={"ID":"b8cab9b4-b072-42f9-9c57-1986f36fd954","Type":"ContainerStarted","Data":"d79eb3fc2bd95bc1a0e0eedeb271737b6066aa450763e2716d56be37380b453c"}
Sep 18 19:45:47.715480 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:47.715446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m" event={"ID":"b8cab9b4-b072-42f9-9c57-1986f36fd954","Type":"ContainerStarted","Data":"ff6a16e1bf311e8a26ffab54500bb017e0f67fe90dd45c0a8c2713f7afcae6da"}
Sep 18 19:45:47.715863 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:47.715518    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:45:47.730084 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:47.730042    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m" podStartSLOduration=2.123367078 podStartE2EDuration="4.730029435s" totalImagesPullingTime="2.606662357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:47.729484669 +0000 UTC m=+441.983691287" watchObservedRunningTime="2026-09-18 19:45:47.730029435 +0000 UTC m=+441.984236039"
Sep 18 19:45:58.720670 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:45:58.720638    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-9pb2m"
Sep 18 19:46:13.322901 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.322869    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"]
Sep 18 19:46:13.325924 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.325909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.328587 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.328563    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mzbnz\""
Sep 18 19:46:13.328691 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.328659    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 19:46:13.328783 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.328758    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 19:46:13.344173 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.344148    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"]
Sep 18 19:46:13.435922 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.435887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfbfk\" (UniqueName: \"kubernetes.io/projected/ea16aa43-9fea-4f01-821f-c3fe02e87a7e-kube-api-access-hfbfk\") pod \"kuadrant-operator-controller-manager-6d4cb79c-sl7nz\" (UID: \"ea16aa43-9fea-4f01-821f-c3fe02e87a7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.436064 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.435939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea16aa43-9fea-4f01-821f-c3fe02e87a7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-sl7nz\" (UID: \"ea16aa43-9fea-4f01-821f-c3fe02e87a7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.537183 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.537152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea16aa43-9fea-4f01-821f-c3fe02e87a7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-sl7nz\" (UID: \"ea16aa43-9fea-4f01-821f-c3fe02e87a7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.537309 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.537235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfbfk\" (UniqueName: \"kubernetes.io/projected/ea16aa43-9fea-4f01-821f-c3fe02e87a7e-kube-api-access-hfbfk\") pod \"kuadrant-operator-controller-manager-6d4cb79c-sl7nz\" (UID: \"ea16aa43-9fea-4f01-821f-c3fe02e87a7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.537547 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.537527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea16aa43-9fea-4f01-821f-c3fe02e87a7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-sl7nz\" (UID: \"ea16aa43-9fea-4f01-821f-c3fe02e87a7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.545521 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.545502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfbfk\" (UniqueName: \"kubernetes.io/projected/ea16aa43-9fea-4f01-821f-c3fe02e87a7e-kube-api-access-hfbfk\") pod \"kuadrant-operator-controller-manager-6d4cb79c-sl7nz\" (UID: \"ea16aa43-9fea-4f01-821f-c3fe02e87a7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.635387 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.635309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:13.909127 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:13.909045    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"]
Sep 18 19:46:13.912175 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:46:13.912146    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea16aa43_9fea_4f01_821f_c3fe02e87a7e.slice/crio-cbed579c1bf09635ac99d735e43a6da788641f6762d2544274cdd7180e6291e9 WatchSource:0}: Error finding container cbed579c1bf09635ac99d735e43a6da788641f6762d2544274cdd7180e6291e9: Status 404 returned error can't find the container with id cbed579c1bf09635ac99d735e43a6da788641f6762d2544274cdd7180e6291e9
Sep 18 19:46:14.812166 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:14.812127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz" event={"ID":"ea16aa43-9fea-4f01-821f-c3fe02e87a7e","Type":"ContainerStarted","Data":"cbed579c1bf09635ac99d735e43a6da788641f6762d2544274cdd7180e6291e9"}
Sep 18 19:46:18.830069 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:18.830029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz" event={"ID":"ea16aa43-9fea-4f01-821f-c3fe02e87a7e","Type":"ContainerStarted","Data":"7489bc892d8487002a589b405ffa21a31edbd9e5488cd1b6788892e2092ced5f"}
Sep 18 19:46:18.830489 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:18.830131    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:46:18.852474 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:18.852425    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz" podStartSLOduration=1.759339314 podStartE2EDuration="5.852407128s" totalImagesPullingTime="4.093067814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:18.850373683 +0000 UTC m=+473.104580284" watchObservedRunningTime="2026-09-18 19:46:18.852407128 +0000 UTC m=+473.106613734"
Sep 18 19:46:29.836150 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:46:29.836117    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-sl7nz"
Sep 18 19:47:03.952896 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:03.952865    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"]
Sep 18 19:47:03.957129 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:03.957106    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:03.959472 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:03.959453    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jkpnw\""
Sep 18 19:47:03.959591 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:03.959453    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 19:47:03.967690 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:03.967664    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"]
Sep 18 19:47:03.991026 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:03.990997    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"]
Sep 18 19:47:04.097246 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.097211    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52157160-3c1d-4d59-88f9-2ea8f2e1e285-config-file\") pod \"limitador-limitador-84994f7ffc-tw5h6\" (UID: \"52157160-3c1d-4d59-88f9-2ea8f2e1e285\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:04.097471 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.097334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/52157160-3c1d-4d59-88f9-2ea8f2e1e285-kube-api-access-8dl2q\") pod \"limitador-limitador-84994f7ffc-tw5h6\" (UID: \"52157160-3c1d-4d59-88f9-2ea8f2e1e285\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:04.198032 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.198003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52157160-3c1d-4d59-88f9-2ea8f2e1e285-config-file\") pod \"limitador-limitador-84994f7ffc-tw5h6\" (UID: \"52157160-3c1d-4d59-88f9-2ea8f2e1e285\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:04.198179 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.198047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/52157160-3c1d-4d59-88f9-2ea8f2e1e285-kube-api-access-8dl2q\") pod \"limitador-limitador-84994f7ffc-tw5h6\" (UID: \"52157160-3c1d-4d59-88f9-2ea8f2e1e285\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:04.198680 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.198655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52157160-3c1d-4d59-88f9-2ea8f2e1e285-config-file\") pod \"limitador-limitador-84994f7ffc-tw5h6\" (UID: \"52157160-3c1d-4d59-88f9-2ea8f2e1e285\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:04.205342 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.205291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/52157160-3c1d-4d59-88f9-2ea8f2e1e285-kube-api-access-8dl2q\") pod \"limitador-limitador-84994f7ffc-tw5h6\" (UID: \"52157160-3c1d-4d59-88f9-2ea8f2e1e285\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:04.268329 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.268301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:04.554938 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.554915    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"]
Sep 18 19:47:04.557516 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:47:04.557491    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52157160_3c1d_4d59_88f9_2ea8f2e1e285.slice/crio-4d076198582b884e632733f84b35b1b35aa692d9065de592607d7aac91d7edeb WatchSource:0}: Error finding container 4d076198582b884e632733f84b35b1b35aa692d9065de592607d7aac91d7edeb: Status 404 returned error can't find the container with id 4d076198582b884e632733f84b35b1b35aa692d9065de592607d7aac91d7edeb
Sep 18 19:47:04.697967 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.697939    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g2l8"]
Sep 18 19:47:04.700570 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.700553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g2l8"
Sep 18 19:47:04.702822 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.702804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9twvg\""
Sep 18 19:47:04.705922 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.705888    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g2l8"]
Sep 18 19:47:04.803056 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.803029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwjs\" (UniqueName: \"kubernetes.io/projected/94511a68-be7d-4cbd-acab-22ee609fef0a-kube-api-access-8vwjs\") pod \"authorino-59c974bd48-9g2l8\" (UID: \"94511a68-be7d-4cbd-acab-22ee609fef0a\") " pod="kuadrant-system/authorino-59c974bd48-9g2l8"
Sep 18 19:47:04.903950 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.903881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwjs\" (UniqueName: \"kubernetes.io/projected/94511a68-be7d-4cbd-acab-22ee609fef0a-kube-api-access-8vwjs\") pod \"authorino-59c974bd48-9g2l8\" (UID: \"94511a68-be7d-4cbd-acab-22ee609fef0a\") " pod="kuadrant-system/authorino-59c974bd48-9g2l8"
Sep 18 19:47:04.912056 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.912031    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwjs\" (UniqueName: \"kubernetes.io/projected/94511a68-be7d-4cbd-acab-22ee609fef0a-kube-api-access-8vwjs\") pod \"authorino-59c974bd48-9g2l8\" (UID: \"94511a68-be7d-4cbd-acab-22ee609fef0a\") " pod="kuadrant-system/authorino-59c974bd48-9g2l8"
Sep 18 19:47:04.993271 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:04.993235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6" event={"ID":"52157160-3c1d-4d59-88f9-2ea8f2e1e285","Type":"ContainerStarted","Data":"4d076198582b884e632733f84b35b1b35aa692d9065de592607d7aac91d7edeb"}
Sep 18 19:47:05.010435 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:05.010414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g2l8"
Sep 18 19:47:05.275691 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:05.275656    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g2l8"]
Sep 18 19:47:05.442493 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:47:05.442466    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94511a68_be7d_4cbd_acab_22ee609fef0a.slice/crio-cc04a787407d3bc4db8d1cf27c6154ddf1a64204c39e360273394effa5353d1e WatchSource:0}: Error finding container cc04a787407d3bc4db8d1cf27c6154ddf1a64204c39e360273394effa5353d1e: Status 404 returned error can't find the container with id cc04a787407d3bc4db8d1cf27c6154ddf1a64204c39e360273394effa5353d1e
Sep 18 19:47:05.997439 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:05.997400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g2l8" event={"ID":"94511a68-be7d-4cbd-acab-22ee609fef0a","Type":"ContainerStarted","Data":"cc04a787407d3bc4db8d1cf27c6154ddf1a64204c39e360273394effa5353d1e"}
Sep 18 19:47:05.998873 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:05.998850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6" event={"ID":"52157160-3c1d-4d59-88f9-2ea8f2e1e285","Type":"ContainerStarted","Data":"54152563596b5246cb67f1b747cb6257e3a0b4a464db15efa4740c20cd4b10c7"}
Sep 18 19:47:05.999004 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:05.998990    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:06.016300 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:06.016257    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6" podStartSLOduration=2.105868929 podStartE2EDuration="3.016243584s" totalImagesPullingTime="910.374655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:06.014688856 +0000 UTC m=+520.268895463" watchObservedRunningTime="2026-09-18 19:47:06.016243584 +0000 UTC m=+520.270450188"
Sep 18 19:47:10.015394 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:10.015340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g2l8" event={"ID":"94511a68-be7d-4cbd-acab-22ee609fef0a","Type":"ContainerStarted","Data":"b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd"}
Sep 18 19:47:10.029059 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:10.029012    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-9g2l8" podStartSLOduration=2.006289857 podStartE2EDuration="6.028997272s" totalImagesPullingTime="4.022707415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:10.028918507 +0000 UTC m=+524.283125126" watchObservedRunningTime="2026-09-18 19:47:10.028997272 +0000 UTC m=+524.283203876"
Sep 18 19:47:17.004096 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:17.004064    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw5h6"
Sep 18 19:47:27.152086 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:27.152045    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g2l8"]
Sep 18 19:47:27.152580 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:27.152218    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-9g2l8" podUID="94511a68-be7d-4cbd-acab-22ee609fef0a" containerName="authorino" containerID="cri-o://b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd" gracePeriod=30
Sep 18 19:47:27.382296 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:27.382273    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g2l8"
Sep 18 19:47:27.466013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:27.465946    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vwjs\" (UniqueName: \"kubernetes.io/projected/94511a68-be7d-4cbd-acab-22ee609fef0a-kube-api-access-8vwjs\") pod \"94511a68-be7d-4cbd-acab-22ee609fef0a\" (UID: \"94511a68-be7d-4cbd-acab-22ee609fef0a\") "
Sep 18 19:47:27.468217 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:27.468179    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94511a68-be7d-4cbd-acab-22ee609fef0a-kube-api-access-8vwjs" (OuterVolumeSpecName: "kube-api-access-8vwjs") pod "94511a68-be7d-4cbd-acab-22ee609fef0a" (UID: "94511a68-be7d-4cbd-acab-22ee609fef0a"). InnerVolumeSpecName "kube-api-access-8vwjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:27.567014 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:27.566977    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwjs\" (UniqueName: \"kubernetes.io/projected/94511a68-be7d-4cbd-acab-22ee609fef0a-kube-api-access-8vwjs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:47:28.087205 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.087169    2583 generic.go:356] "Generic (PLEG): container finished" podID="94511a68-be7d-4cbd-acab-22ee609fef0a" containerID="b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd" exitCode=0
Sep 18 19:47:28.087344 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.087209    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g2l8"
Sep 18 19:47:28.087344 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.087252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g2l8" event={"ID":"94511a68-be7d-4cbd-acab-22ee609fef0a","Type":"ContainerDied","Data":"b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd"}
Sep 18 19:47:28.087344 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.087288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g2l8" event={"ID":"94511a68-be7d-4cbd-acab-22ee609fef0a","Type":"ContainerDied","Data":"cc04a787407d3bc4db8d1cf27c6154ddf1a64204c39e360273394effa5353d1e"}
Sep 18 19:47:28.087344 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.087304    2583 scope.go:117] "RemoveContainer" containerID="b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd"
Sep 18 19:47:28.094817 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.094803    2583 scope.go:117] "RemoveContainer" containerID="b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd"
Sep 18 19:47:28.095047 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:47:28.095028    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd\": container with ID starting with b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd not found: ID does not exist" containerID="b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd"
Sep 18 19:47:28.095090 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.095055    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd"} err="failed to get container status \"b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd\": rpc error: code = NotFound desc = could not find container \"b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd\": container with ID starting with b4bcfc16fa59bb0f697ee317f3331f657fd1d09d88e313309cfeb23a80b587cd not found: ID does not exist"
Sep 18 19:47:28.103153 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.103135    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g2l8"]
Sep 18 19:47:28.106732 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:28.106716    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g2l8"]
Sep 18 19:47:29.965780 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:47:29.965751    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94511a68-be7d-4cbd-acab-22ee609fef0a" path="/var/lib/kubelet/pods/94511a68-be7d-4cbd-acab-22ee609fef0a/volumes"
Sep 18 19:48:27.872550 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:48:27.872528    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:48:27.873092 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:48:27.873073    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:49:27.998289 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:27.998256    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-j8fxj"]
Sep 18 19:49:27.999954 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:27.998662    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94511a68-be7d-4cbd-acab-22ee609fef0a" containerName="authorino"
Sep 18 19:49:27.999954 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:27.998680    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94511a68-be7d-4cbd-acab-22ee609fef0a" containerName="authorino"
Sep 18 19:49:27.999954 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:27.998739    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="94511a68-be7d-4cbd-acab-22ee609fef0a" containerName="authorino"
Sep 18 19:49:28.000435 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.000419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.002837 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.002819    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:49:28.002935 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.002874    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:49:28.003817 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.003799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:49:28.003936 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.003807    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-n7prf\""
Sep 18 19:49:28.003936 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.003873    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:49:28.008631 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.008583    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-j8fxj"]
Sep 18 19:49:28.119343 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.119316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfvnf\" (UniqueName: \"kubernetes.io/projected/eea59772-56aa-4043-a6a3-e75d99fb3162-kube-api-access-cfvnf\") pod \"s3-init-j8fxj\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") " pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.119518 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.119380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea59772-56aa-4043-a6a3-e75d99fb3162-model-cache\") pod \"s3-init-j8fxj\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") " pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.220762 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.220723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea59772-56aa-4043-a6a3-e75d99fb3162-model-cache\") pod \"s3-init-j8fxj\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") " pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.220937 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.220795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfvnf\" (UniqueName: \"kubernetes.io/projected/eea59772-56aa-4043-a6a3-e75d99fb3162-kube-api-access-cfvnf\") pod \"s3-init-j8fxj\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") " pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.221191 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.221172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea59772-56aa-4043-a6a3-e75d99fb3162-model-cache\") pod \"s3-init-j8fxj\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") " pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.229036 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.229014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfvnf\" (UniqueName: \"kubernetes.io/projected/eea59772-56aa-4043-a6a3-e75d99fb3162-kube-api-access-cfvnf\") pod \"s3-init-j8fxj\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") " pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.310752 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.310723    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8fxj"
Sep 18 19:49:28.579636 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:28.579568    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-j8fxj"]
Sep 18 19:49:28.582697 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:49:28.582669    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeea59772_56aa_4043_a6a3_e75d99fb3162.slice/crio-4e471c7d5ae47ed3a5f389dcb5b11b0c1b4f13e8a867168c001da5b007739716 WatchSource:0}: Error finding container 4e471c7d5ae47ed3a5f389dcb5b11b0c1b4f13e8a867168c001da5b007739716: Status 404 returned error can't find the container with id 4e471c7d5ae47ed3a5f389dcb5b11b0c1b4f13e8a867168c001da5b007739716
Sep 18 19:49:29.550510 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:29.550473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8fxj" event={"ID":"eea59772-56aa-4043-a6a3-e75d99fb3162","Type":"ContainerStarted","Data":"4e471c7d5ae47ed3a5f389dcb5b11b0c1b4f13e8a867168c001da5b007739716"}
Sep 18 19:49:32.565977 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:32.565942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8fxj" event={"ID":"eea59772-56aa-4043-a6a3-e75d99fb3162","Type":"ContainerStarted","Data":"f49a41c655a6e6eb5a6a70039b85d3d833a4b31a38d6795aa104328e5f2de762"}
Sep 18 19:49:33.572136 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:33.572102    2583 generic.go:356] "Generic (PLEG): container finished" podID="eea59772-56aa-4043-a6a3-e75d99fb3162" containerID="f49a41c655a6e6eb5a6a70039b85d3d833a4b31a38d6795aa104328e5f2de762" exitCode=0
Sep 18 19:49:33.572443 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:33.572146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8fxj" event={"ID":"eea59772-56aa-4043-a6a3-e75d99fb3162","Type":"ContainerDied","Data":"f49a41c655a6e6eb5a6a70039b85d3d833a4b31a38d6795aa104328e5f2de762"}
Sep 18 19:49:38.597645 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:38.597602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8fxj" event={"ID":"eea59772-56aa-4043-a6a3-e75d99fb3162","Type":"ContainerStarted","Data":"9fb3e0aaae9388cd7b57e399c930b9e8931a110cb7d0624d245530047f733201"}
Sep 18 19:49:38.615899 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:49:38.615846    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-j8fxj" podStartSLOduration=2.332429401 podStartE2EDuration="11.615829911s" totalImagesPullingTime="7.784484576s" totalInitContainerRuntime="1.498915934s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:38.615267523 +0000 UTC m=+672.869474128" watchObservedRunningTime="2026-09-18 19:49:38.615829911 +0000 UTC m=+672.870036515"
Sep 18 19:50:42.856010 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:42.855979    2583 generic.go:356] "Generic (PLEG): container finished" podID="eea59772-56aa-4043-a6a3-e75d99fb3162" containerID="9fb3e0aaae9388cd7b57e399c930b9e8931a110cb7d0624d245530047f733201" exitCode=0
Sep 18 19:50:42.856339 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:42.856048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8fxj" event={"ID":"eea59772-56aa-4043-a6a3-e75d99fb3162","Type":"ContainerDied","Data":"9fb3e0aaae9388cd7b57e399c930b9e8931a110cb7d0624d245530047f733201"}
Sep 18 19:50:43.985419 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:43.985392    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8fxj"
Sep 18 19:50:44.074258 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.074228    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea59772-56aa-4043-a6a3-e75d99fb3162-model-cache\") pod \"eea59772-56aa-4043-a6a3-e75d99fb3162\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") "
Sep 18 19:50:44.074428 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.074337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfvnf\" (UniqueName: \"kubernetes.io/projected/eea59772-56aa-4043-a6a3-e75d99fb3162-kube-api-access-cfvnf\") pod \"eea59772-56aa-4043-a6a3-e75d99fb3162\" (UID: \"eea59772-56aa-4043-a6a3-e75d99fb3162\") "
Sep 18 19:50:44.076684 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.076652    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea59772-56aa-4043-a6a3-e75d99fb3162-kube-api-access-cfvnf" (OuterVolumeSpecName: "kube-api-access-cfvnf") pod "eea59772-56aa-4043-a6a3-e75d99fb3162" (UID: "eea59772-56aa-4043-a6a3-e75d99fb3162"). InnerVolumeSpecName "kube-api-access-cfvnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:50:44.092583 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.092557    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea59772-56aa-4043-a6a3-e75d99fb3162-model-cache" (OuterVolumeSpecName: "model-cache") pod "eea59772-56aa-4043-a6a3-e75d99fb3162" (UID: "eea59772-56aa-4043-a6a3-e75d99fb3162"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:50:44.175103 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.175034    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfvnf\" (UniqueName: \"kubernetes.io/projected/eea59772-56aa-4043-a6a3-e75d99fb3162-kube-api-access-cfvnf\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:50:44.175103 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.175057    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea59772-56aa-4043-a6a3-e75d99fb3162-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:50:44.865235 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.865199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8fxj" event={"ID":"eea59772-56aa-4043-a6a3-e75d99fb3162","Type":"ContainerDied","Data":"4e471c7d5ae47ed3a5f389dcb5b11b0c1b4f13e8a867168c001da5b007739716"}
Sep 18 19:50:44.865235 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.865226    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8fxj"
Sep 18 19:50:44.865485 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:50:44.865234    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e471c7d5ae47ed3a5f389dcb5b11b0c1b4f13e8a867168c001da5b007739716"
Sep 18 19:51:12.522905 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.522862    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"]
Sep 18 19:51:12.523469 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.523253    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea59772-56aa-4043-a6a3-e75d99fb3162" containerName="download-hf-model"
Sep 18 19:51:12.523469 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.523271    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea59772-56aa-4043-a6a3-e75d99fb3162" containerName="download-hf-model"
Sep 18 19:51:12.523469 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.523302    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea59772-56aa-4043-a6a3-e75d99fb3162" containerName="s3-init"
Sep 18 19:51:12.523469 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.523311    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea59772-56aa-4043-a6a3-e75d99fb3162" containerName="s3-init"
Sep 18 19:51:12.523469 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.523395    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea59772-56aa-4043-a6a3-e75d99fb3162" containerName="s3-init"
Sep 18 19:51:12.526537 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.526516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.529233 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.529202    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 18 19:51:12.529384 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.529202    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:51:12.529485 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.529463    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 18 19:51:12.529644 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.529546    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 18 19:51:12.529644 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.529571    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 18 19:51:12.529644 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.529576    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-tfzqv\""
Sep 18 19:51:12.537938 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.537919    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"]
Sep 18 19:51:12.579610 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee35c3ef-f7b9-4a46-b386-5406285625c8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.579715 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee35c3ef-f7b9-4a46-b386-5406285625c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.579715 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.579715 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.579715 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.579884 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.579884 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkbq\" (UniqueName: \"kubernetes.io/projected/ee35c3ef-f7b9-4a46-b386-5406285625c8-kube-api-access-pkkbq\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.579884 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.579802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.680926 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.680901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681024 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.680929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkbq\" (UniqueName: \"kubernetes.io/projected/ee35c3ef-f7b9-4a46-b386-5406285625c8-kube-api-access-pkkbq\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681024 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.680960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681024 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.680990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee35c3ef-f7b9-4a46-b386-5406285625c8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681024 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee35c3ef-f7b9-4a46-b386-5406285625c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681207 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681207 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681307 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681382 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681446 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681503 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681606 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.681820 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.681804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee35c3ef-f7b9-4a46-b386-5406285625c8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.683572 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.683550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.683775 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.683757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee35c3ef-f7b9-4a46-b386-5406285625c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.689642 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.689617    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkbq\" (UniqueName: \"kubernetes.io/projected/ee35c3ef-f7b9-4a46-b386-5406285625c8-kube-api-access-pkkbq\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:12.838599 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:12.838538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:13.113731 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:13.113650    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"]
Sep 18 19:51:13.116294 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:51:13.116261    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee35c3ef_f7b9_4a46_b386_5406285625c8.slice/crio-ed0ca44b573b854ca673dc47bae75c2ffadd5449661a4bff0520cdfb65a0c981 WatchSource:0}: Error finding container ed0ca44b573b854ca673dc47bae75c2ffadd5449661a4bff0520cdfb65a0c981: Status 404 returned error can't find the container with id ed0ca44b573b854ca673dc47bae75c2ffadd5449661a4bff0520cdfb65a0c981
Sep 18 19:51:13.978691 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:13.978646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" event={"ID":"ee35c3ef-f7b9-4a46-b386-5406285625c8","Type":"ContainerStarted","Data":"8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427"}
Sep 18 19:51:13.979168 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:13.978712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" event={"ID":"ee35c3ef-f7b9-4a46-b386-5406285625c8","Type":"ContainerStarted","Data":"ed0ca44b573b854ca673dc47bae75c2ffadd5449661a4bff0520cdfb65a0c981"}
Sep 18 19:51:15.990910 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:15.990872    2583 generic.go:356] "Generic (PLEG): container finished" podID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerID="8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427" exitCode=0
Sep 18 19:51:15.991276 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:15.990953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" event={"ID":"ee35c3ef-f7b9-4a46-b386-5406285625c8","Type":"ContainerDied","Data":"8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427"}
Sep 18 19:51:15.992335 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:15.992322    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:51:44.137561 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:44.137522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" event={"ID":"ee35c3ef-f7b9-4a46-b386-5406285625c8","Type":"ContainerStarted","Data":"cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4"}
Sep 18 19:51:44.161448 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:44.161386    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podStartSLOduration=2.817875264 podStartE2EDuration="32.16134425s" totalImagesPullingTime="27.255015729s" totalInitContainerRuntime="2.088453257s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:51:44.157994332 +0000 UTC m=+798.412200936" watchObservedRunningTime="2026-09-18 19:51:44.16134425 +0000 UTC m=+798.415550862"
Sep 18 19:51:44.839017 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:44.838982    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:52.838788 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:52.838752    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:51:52.840497 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:51:52.840468    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:52:02.839955 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:52:02.839906    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:52:12.839701 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:52:12.839661    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:52:22.839160 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:52:22.839109    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:52:32.839072 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:52:32.839028    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:52:42.839563 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:52:42.839508    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:52:52.839378 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:52:52.839311    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:53:02.839525 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:02.839479    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:53:12.839226 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:12.839145    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:53:22.839956 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:22.839916    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": dial tcp 10.134.0.21:8000: connect: connection refused"
Sep 18 19:53:27.900177 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:27.900139    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:53:27.903178 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:27.903157    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:53:32.850919 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:32.850889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:53:32.858557 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:32.858535    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:53:38.238626 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:38.238595    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"]
Sep 18 19:53:53.260998 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.260953    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" containerID="cri-o://cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4" gracePeriod=45
Sep 18 19:53:53.915259 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.915232    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:53:53.975702 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975662    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkkbq\" (UniqueName: \"kubernetes.io/projected/ee35c3ef-f7b9-4a46-b386-5406285625c8-kube-api-access-pkkbq\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.975835 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975714    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-kserve-provision-location\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.975835 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975732    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-home\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.975835 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee35c3ef-f7b9-4a46-b386-5406285625c8-cabundle-cert\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.975835 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975779    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-model-cache\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.975835 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975807    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee35c3ef-f7b9-4a46-b386-5406285625c8-tls-certs\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.976131 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-dshm\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.976131 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.975890    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-tmp-dir\") pod \"ee35c3ef-f7b9-4a46-b386-5406285625c8\" (UID: \"ee35c3ef-f7b9-4a46-b386-5406285625c8\") "
Sep 18 19:53:53.976233 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.976133    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:53.976233 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.976190    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee35c3ef-f7b9-4a46-b386-5406285625c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:53.976536 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.976507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-home" (OuterVolumeSpecName: "home") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:53.978144 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.978118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee35c3ef-f7b9-4a46-b386-5406285625c8-kube-api-access-pkkbq" (OuterVolumeSpecName: "kube-api-access-pkkbq") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "kube-api-access-pkkbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:53.978332 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.978300    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-dshm" (OuterVolumeSpecName: "dshm") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:53.978648 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.978630    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee35c3ef-f7b9-4a46-b386-5406285625c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:53.988908 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.988884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:53.995534 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:53.995514    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee35c3ef-f7b9-4a46-b386-5406285625c8" (UID: "ee35c3ef-f7b9-4a46-b386-5406285625c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:54.077156 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077105    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.077156 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077144    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.077156 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077155    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkkbq\" (UniqueName: \"kubernetes.io/projected/ee35c3ef-f7b9-4a46-b386-5406285625c8-kube-api-access-pkkbq\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.077285 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077165    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.077285 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077173    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.077285 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077180    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee35c3ef-f7b9-4a46-b386-5406285625c8-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.077285 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077188    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee35c3ef-f7b9-4a46-b386-5406285625c8-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.077285 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.077195    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee35c3ef-f7b9-4a46-b386-5406285625c8-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:53:54.686058 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.686026    2583 generic.go:356] "Generic (PLEG): container finished" podID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerID="cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4" exitCode=0
Sep 18 19:53:54.686544 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.686093    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"
Sep 18 19:53:54.686544 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.686101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" event={"ID":"ee35c3ef-f7b9-4a46-b386-5406285625c8","Type":"ContainerDied","Data":"cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4"}
Sep 18 19:53:54.686544 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.686156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" event={"ID":"ee35c3ef-f7b9-4a46-b386-5406285625c8","Type":"ContainerDied","Data":"ed0ca44b573b854ca673dc47bae75c2ffadd5449661a4bff0520cdfb65a0c981"}
Sep 18 19:53:54.686544 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.686191    2583 scope.go:117] "RemoveContainer" containerID="cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4"
Sep 18 19:53:54.696469 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.696434    2583 scope.go:117] "RemoveContainer" containerID="8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427"
Sep 18 19:53:54.704933 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.704910    2583 scope.go:117] "RemoveContainer" containerID="cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4"
Sep 18 19:53:54.705265 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:53:54.705245    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4\": container with ID starting with cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4 not found: ID does not exist" containerID="cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4"
Sep 18 19:53:54.705376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.705273    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4"} err="failed to get container status \"cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4\": rpc error: code = NotFound desc = could not find container \"cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4\": container with ID starting with cc92468aa5602d022b83c548321bb48dc456109de744b5e5b0328f58d9a1e5f4 not found: ID does not exist"
Sep 18 19:53:54.705376 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.705289    2583 scope.go:117] "RemoveContainer" containerID="8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427"
Sep 18 19:53:54.705598 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:53:54.705580    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427\": container with ID starting with 8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427 not found: ID does not exist" containerID="8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427"
Sep 18 19:53:54.705691 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.705605    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427"} err="failed to get container status \"8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427\": rpc error: code = NotFound desc = could not find container \"8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427\": container with ID starting with 8531f793390172d227457a0b6de1dcb75cec3f31b33280645f178d21e05e8427 not found: ID does not exist"
Sep 18 19:53:54.706750 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.706728    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"]
Sep 18 19:53:54.709674 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.709654    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f"]
Sep 18 19:53:54.852599 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:54.852568    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c79745964-7446f" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.21:8000/health\": context deadline exceeded"
Sep 18 19:53:55.965907 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:53:55.965874    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" path="/var/lib/kubelet/pods/ee35c3ef-f7b9-4a46-b386-5406285625c8/volumes"
Sep 18 19:54:09.110830 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.110800    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 19:54:09.111246 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.111140    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="storage-initializer"
Sep 18 19:54:09.111246 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.111156    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="storage-initializer"
Sep 18 19:54:09.111246 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.111182    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main"
Sep 18 19:54:09.111246 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.111190    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main"
Sep 18 19:54:09.111429 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.111258    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee35c3ef-f7b9-4a46-b386-5406285625c8" containerName="main"
Sep 18 19:54:09.116189 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.116169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.118891 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.118871    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:54:09.119329 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.119313    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 19:54:09.119555 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.119538    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 19:54:09.120154 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.120136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 19:54:09.121248 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.121229    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 19:54:09.121377 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.121231    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-nlqwt\""
Sep 18 19:54:09.121443 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.121382    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-68qj6\""
Sep 18 19:54:09.132309 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.132280    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 19:54:09.185091 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.185251 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.185251 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35ddbf05-f98e-4789-819e-e2595b65e4a6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.185394 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.185394 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wv86\" (UniqueName: \"kubernetes.io/projected/35ddbf05-f98e-4789-819e-e2595b65e4a6-kube-api-access-2wv86\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.185394 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ddbf05-f98e-4789-819e-e2595b65e4a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.185526 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.185526 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.185449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.286604 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35ddbf05-f98e-4789-819e-e2595b65e4a6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.286604 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.286918 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wv86\" (UniqueName: \"kubernetes.io/projected/35ddbf05-f98e-4789-819e-e2595b65e4a6-kube-api-access-2wv86\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.286918 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ddbf05-f98e-4789-819e-e2595b65e4a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.286918 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.286918 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.287135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.287135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.286974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.287135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.287005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.287135 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.287088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.287323 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.287218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35ddbf05-f98e-4789-819e-e2595b65e4a6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.287323 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.287263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.287323 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.287267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.289212 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.289192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.289407 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.289391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ddbf05-f98e-4789-819e-e2595b65e4a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.306042 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.306021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wv86\" (UniqueName: \"kubernetes.io/projected/35ddbf05-f98e-4789-819e-e2595b65e4a6-kube-api-access-2wv86\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.427527 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.427463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:09.701465 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.701389    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 19:54:09.704668 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:54:09.704645    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ddbf05_f98e_4789_819e_e2595b65e4a6.slice/crio-4a198a0c193eb795e041f83f9e7fa883fef8a59084102c06c874cb9dc96c1ec9 WatchSource:0}: Error finding container 4a198a0c193eb795e041f83f9e7fa883fef8a59084102c06c874cb9dc96c1ec9: Status 404 returned error can't find the container with id 4a198a0c193eb795e041f83f9e7fa883fef8a59084102c06c874cb9dc96c1ec9
Sep 18 19:54:09.749180 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:09.749157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"35ddbf05-f98e-4789-819e-e2595b65e4a6","Type":"ContainerStarted","Data":"4a198a0c193eb795e041f83f9e7fa883fef8a59084102c06c874cb9dc96c1ec9"}
Sep 18 19:54:10.758208 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:10.758167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"35ddbf05-f98e-4789-819e-e2595b65e4a6","Type":"ContainerStarted","Data":"101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54"}
Sep 18 19:54:12.768427 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:12.768397    2583 generic.go:356] "Generic (PLEG): container finished" podID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerID="101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54" exitCode=0
Sep 18 19:54:12.768759 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:12.768465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"35ddbf05-f98e-4789-819e-e2595b65e4a6","Type":"ContainerDied","Data":"101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54"}
Sep 18 19:54:13.774547 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:13.774513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"35ddbf05-f98e-4789-819e-e2595b65e4a6","Type":"ContainerStarted","Data":"9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423"}
Sep 18 19:54:13.795550 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:13.795495    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.700598785 podStartE2EDuration="4.795477582s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.094878797s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:13.79408669 +0000 UTC m=+948.048293319" watchObservedRunningTime="2026-09-18 19:54:13.795477582 +0000 UTC m=+948.049684186"
Sep 18 19:54:14.427776 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:14.427739    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:19.428372 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:19.428316    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:54:19.429560 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:19.429534    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:54:29.428860 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:29.428817    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:54:39.428546 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:39.428448    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:54:49.428051 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:49.428013    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:54:59.428031 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:54:59.427988    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:55:09.428640 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:55:09.428604    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:55:19.428498 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:55:19.428449    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:55:29.428015 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:55:29.427972    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:55:39.428372 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:55:39.428327    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:55:49.437783 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:55:49.437748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:55:49.446272 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:55:49.446251    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:56:03.229341 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:03.229302    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 19:56:18.253461 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:18.253401    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" containerID="cri-o://9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423" gracePeriod=45
Sep 18 19:56:18.439400 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:18.439333    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.22:8000/health\": dial tcp 10.134.0.22:8000: connect: connection refused"
Sep 18 19:56:18.892142 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:18.892121    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:56:19.011701 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011674    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-kserve-provision-location\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.011872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-model-cache\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.011872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-dshm\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.011872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-home\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.011872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011776    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35ddbf05-f98e-4789-819e-e2595b65e4a6-cabundle-cert\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.011872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011818    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wv86\" (UniqueName: \"kubernetes.io/projected/35ddbf05-f98e-4789-819e-e2595b65e4a6-kube-api-access-2wv86\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.011872 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-tmp-dir\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.012183 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.011877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ddbf05-f98e-4789-819e-e2595b65e4a6-tls-certs\") pod \"35ddbf05-f98e-4789-819e-e2595b65e4a6\" (UID: \"35ddbf05-f98e-4789-819e-e2595b65e4a6\") "
Sep 18 19:56:19.012183 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.012039    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:19.012183 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.012154    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.012183 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.012170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ddbf05-f98e-4789-819e-e2595b65e4a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:19.012940 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.012909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-home" (OuterVolumeSpecName: "home") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:19.014269 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.014223    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-dshm" (OuterVolumeSpecName: "dshm") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:19.014645 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.014622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ddbf05-f98e-4789-819e-e2595b65e4a6-kube-api-access-2wv86" (OuterVolumeSpecName: "kube-api-access-2wv86") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "kube-api-access-2wv86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:19.014957 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.014935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35ddbf05-f98e-4789-819e-e2595b65e4a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:19.030149 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.030126    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:19.035865 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.035835    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35ddbf05-f98e-4789-819e-e2595b65e4a6" (UID: "35ddbf05-f98e-4789-819e-e2595b65e4a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:19.113447 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.113421    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wv86\" (UniqueName: \"kubernetes.io/projected/35ddbf05-f98e-4789-819e-e2595b65e4a6-kube-api-access-2wv86\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.113447 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.113446    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.113614 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.113456    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ddbf05-f98e-4789-819e-e2595b65e4a6-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.113614 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.113464    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.113614 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.113472    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.113614 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.113484    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ddbf05-f98e-4789-819e-e2595b65e4a6-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.113614 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.113495    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35ddbf05-f98e-4789-819e-e2595b65e4a6-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:56:19.303403 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.303302    2583 generic.go:356] "Generic (PLEG): container finished" podID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerID="9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423" exitCode=0
Sep 18 19:56:19.303403 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.303373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"35ddbf05-f98e-4789-819e-e2595b65e4a6","Type":"ContainerDied","Data":"9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423"}
Sep 18 19:56:19.303403 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.303382    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 19:56:19.304022 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.303429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"35ddbf05-f98e-4789-819e-e2595b65e4a6","Type":"ContainerDied","Data":"4a198a0c193eb795e041f83f9e7fa883fef8a59084102c06c874cb9dc96c1ec9"}
Sep 18 19:56:19.304022 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.303461    2583 scope.go:117] "RemoveContainer" containerID="9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423"
Sep 18 19:56:19.317310 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.317292    2583 scope.go:117] "RemoveContainer" containerID="101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54"
Sep 18 19:56:19.326766 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.326734    2583 scope.go:117] "RemoveContainer" containerID="9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423"
Sep 18 19:56:19.327188 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:56:19.327144    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423\": container with ID starting with 9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423 not found: ID does not exist" containerID="9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423"
Sep 18 19:56:19.327302 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.327200    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423"} err="failed to get container status \"9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423\": rpc error: code = NotFound desc = could not find container \"9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423\": container with ID starting with 9f2524207f3576f61a5836a5167c7186ee427c6174b648d77ad157aac06d7423 not found: ID does not exist"
Sep 18 19:56:19.327302 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.327224    2583 scope.go:117] "RemoveContainer" containerID="101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54"
Sep 18 19:56:19.327630 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:56:19.327603    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54\": container with ID starting with 101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54 not found: ID does not exist" containerID="101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54"
Sep 18 19:56:19.327734 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.327635    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54"} err="failed to get container status \"101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54\": rpc error: code = NotFound desc = could not find container \"101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54\": container with ID starting with 101b3e7265cd9a2bd5652c94cfa0637d6bcd0a685d013ec7c2d3916a64c44a54 not found: ID does not exist"
Sep 18 19:56:19.328711 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.328695    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 19:56:19.332109 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.332087    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 19:56:19.965670 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:56:19.965640    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" path="/var/lib/kubelet/pods/35ddbf05-f98e-4789-819e-e2595b65e4a6/volumes"
Sep 18 19:57:00.612384 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.612332    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"]
Sep 18 19:57:00.612834 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.612659    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main"
Sep 18 19:57:00.612834 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.612671    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main"
Sep 18 19:57:00.612834 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.612685    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="storage-initializer"
Sep 18 19:57:00.612834 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.612690    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="storage-initializer"
Sep 18 19:57:00.612834 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.612750    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ddbf05-f98e-4789-819e-e2595b65e4a6" containerName="main"
Sep 18 19:57:00.615458 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.615426    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.617887 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.617859    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 19:57:00.618913 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.618884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-kd85p\""
Sep 18 19:57:00.619133 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.619115    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 19:57:00.619217 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.619159    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 19:57:00.629277 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.629237    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"]
Sep 18 19:57:00.722255 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.722219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.722449 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.722272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tj4\" (UniqueName: \"kubernetes.io/projected/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-kube-api-access-v6tj4\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.722449 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.722313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.722449 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.722339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.722584 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.722467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.722584 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.722495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.823812 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.823772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.823812 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.823814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.824044 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.823840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.824044 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.823877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tj4\" (UniqueName: \"kubernetes.io/projected/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-kube-api-access-v6tj4\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.824044 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.823911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.824044 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.823936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.824293 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.824272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.824394 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.824306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.824457 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.824413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.826318 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.826295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.826514 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.826495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.831062 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.831040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tj4\" (UniqueName: \"kubernetes.io/projected/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-kube-api-access-v6tj4\") pod \"scheduler-inline-config-test-kserve-8655bc757-gp298\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:00.925252 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:00.925167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:01.194985 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:01.194963    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"]
Sep 18 19:57:01.197628 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:57:01.197593    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0086945c_b48b_4c6e_ad5c_ac1e0016b88d.slice/crio-820bac4f6bc3ce47c1e34f6eb24f7df8a508671bebdda0d456316775ec2c4e0a WatchSource:0}: Error finding container 820bac4f6bc3ce47c1e34f6eb24f7df8a508671bebdda0d456316775ec2c4e0a: Status 404 returned error can't find the container with id 820bac4f6bc3ce47c1e34f6eb24f7df8a508671bebdda0d456316775ec2c4e0a
Sep 18 19:57:01.199678 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:01.199656    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:57:01.478328 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:01.478260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" event={"ID":"0086945c-b48b-4c6e-ad5c-ac1e0016b88d","Type":"ContainerStarted","Data":"820bac4f6bc3ce47c1e34f6eb24f7df8a508671bebdda0d456316775ec2c4e0a"}
Sep 18 19:57:03.488974 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:03.488931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" event={"ID":"0086945c-b48b-4c6e-ad5c-ac1e0016b88d","Type":"ContainerStarted","Data":"4a69e36f5d68cc335b45e6f3297f343d0f1aed94208c1c8ce991cc2eae69d2b7"}
Sep 18 19:57:03.518877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:03.518829    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" podStartSLOduration=1.91197621 podStartE2EDuration="3.518814678s" totalImagesPullingTime="1.606838468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:03.517662645 +0000 UTC m=+1117.771869250" watchObservedRunningTime="2026-09-18 19:57:03.518814678 +0000 UTC m=+1117.773021282"
Sep 18 19:57:03.925654 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:03.925623    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:10.926040 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:10.926001    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:10.938484 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:10.938450    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:11.530920 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:11.530880    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:33.939710 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:33.939672    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"]
Sep 18 19:57:33.949428 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:57:33.949394    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:57:33.949428 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:57:33.949428 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 19:57:33.949428 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="4a69e36f5d68cc335b45e6f3297f343d0f1aed94208c1c8ce991cc2eae69d2b7" cmd=["/bin/sleep","15"]
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:33.949449    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" message="" err=<
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:57:33.949464    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" podUID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" containerName="main" containerID="cri-o://4a69e36f5d68cc335b45e6f3297f343d0f1aed94208c1c8ce991cc2eae69d2b7"
Sep 18 19:57:33.949731 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:33.949521    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" podUID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" containerName="main" containerID="cri-o://4a69e36f5d68cc335b45e6f3297f343d0f1aed94208c1c8ce991cc2eae69d2b7" gracePeriod=60
Sep 18 19:57:34.520754 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.520715    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" podUID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.23:8000/health\": dial tcp 10.134.0.23:8000: connect: connection refused"
Sep 18 19:57:34.617180 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.617152    2583 generic.go:356] "Generic (PLEG): container finished" podID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" containerID="4a69e36f5d68cc335b45e6f3297f343d0f1aed94208c1c8ce991cc2eae69d2b7" exitCode=0
Sep 18 19:57:34.617307 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.617221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" event={"ID":"0086945c-b48b-4c6e-ad5c-ac1e0016b88d","Type":"ContainerDied","Data":"4a69e36f5d68cc335b45e6f3297f343d0f1aed94208c1c8ce991cc2eae69d2b7"}
Sep 18 19:57:34.693264 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.693236    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:34.776294 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-model-cache\") pod \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") "
Sep 18 19:57:34.776294 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776275    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tls-certs\") pod \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") "
Sep 18 19:57:34.776500 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-dshm\") pod \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") "
Sep 18 19:57:34.776500 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776318    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6tj4\" (UniqueName: \"kubernetes.io/projected/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-kube-api-access-v6tj4\") pod \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") "
Sep 18 19:57:34.776500 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-home\") pod \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") "
Sep 18 19:57:34.776500 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776399    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tmp-dir\") pod \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\" (UID: \"0086945c-b48b-4c6e-ad5c-ac1e0016b88d\") "
Sep 18 19:57:34.776704 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776535    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0086945c-b48b-4c6e-ad5c-ac1e0016b88d" (UID: "0086945c-b48b-4c6e-ad5c-ac1e0016b88d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:34.776704 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776614    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:57:34.776704 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0086945c-b48b-4c6e-ad5c-ac1e0016b88d" (UID: "0086945c-b48b-4c6e-ad5c-ac1e0016b88d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:34.776704 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.776695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-home" (OuterVolumeSpecName: "home") pod "0086945c-b48b-4c6e-ad5c-ac1e0016b88d" (UID: "0086945c-b48b-4c6e-ad5c-ac1e0016b88d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:34.778638 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.778616    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-kube-api-access-v6tj4" (OuterVolumeSpecName: "kube-api-access-v6tj4") pod "0086945c-b48b-4c6e-ad5c-ac1e0016b88d" (UID: "0086945c-b48b-4c6e-ad5c-ac1e0016b88d"). InnerVolumeSpecName "kube-api-access-v6tj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:34.778638 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.778620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-dshm" (OuterVolumeSpecName: "dshm") pod "0086945c-b48b-4c6e-ad5c-ac1e0016b88d" (UID: "0086945c-b48b-4c6e-ad5c-ac1e0016b88d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:34.778803 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.778650    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0086945c-b48b-4c6e-ad5c-ac1e0016b88d" (UID: "0086945c-b48b-4c6e-ad5c-ac1e0016b88d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:34.877042 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.877022    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:57:34.877042 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.877042    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:57:34.877176 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.877051    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tj4\" (UniqueName: \"kubernetes.io/projected/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-kube-api-access-v6tj4\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:57:34.877176 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.877059    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:57:34.877176 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:34.877068    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0086945c-b48b-4c6e-ad5c-ac1e0016b88d-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:57:35.622312 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:35.622280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298" event={"ID":"0086945c-b48b-4c6e-ad5c-ac1e0016b88d","Type":"ContainerDied","Data":"820bac4f6bc3ce47c1e34f6eb24f7df8a508671bebdda0d456316775ec2c4e0a"}
Sep 18 19:57:35.622312 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:35.622305    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"
Sep 18 19:57:35.622769 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:35.622350    2583 scope.go:117] "RemoveContainer" containerID="4a69e36f5d68cc335b45e6f3297f343d0f1aed94208c1c8ce991cc2eae69d2b7"
Sep 18 19:57:35.644576 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:35.644547    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"]
Sep 18 19:57:35.650508 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:35.650487    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gp298"]
Sep 18 19:57:35.966118 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:57:35.966032    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" path="/var/lib/kubelet/pods/0086945c-b48b-4c6e-ad5c-ac1e0016b88d/volumes"
Sep 18 19:58:27.925489 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:27.925460    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:58:27.928039 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:27.928013    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 19:58:35.506631 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.506602    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"]
Sep 18 19:58:35.506991 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.506869    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" containerName="main"
Sep 18 19:58:35.506991 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.506879    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" containerName="main"
Sep 18 19:58:35.506991 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.506937    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0086945c-b48b-4c6e-ad5c-ac1e0016b88d" containerName="main"
Sep 18 19:58:35.511030 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.511010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.513400 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.513380    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 19:58:35.514415 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.514387    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 19:58:35.514519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.514464    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-2nc5c\""
Sep 18 19:58:35.514519 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.514464    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 19:58:35.520698 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.520677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"]
Sep 18 19:58:35.617063 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.617037    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.617063 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.617066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j65b\" (UniqueName: \"kubernetes.io/projected/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-kube-api-access-7j65b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.617256 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.617083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.617256 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.617124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.617256 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.617199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.617256 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.617234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.717935 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.717911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718035 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.717938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j65b\" (UniqueName: \"kubernetes.io/projected/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-kube-api-access-7j65b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718035 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.717957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718035 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.718011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718191 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.718052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718191 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.718073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718392 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.718339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718512 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.718377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.718852 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.718748    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.723758 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.721338    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.723758 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.721610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.729253 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.729227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j65b\" (UniqueName: \"kubernetes.io/projected/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-kube-api-access-7j65b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:35.821704 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:35.821674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:36.095058 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:36.095031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"]
Sep 18 19:58:36.097401 ip-10-0-133-25 kubenswrapper[2583]: W0918 19:58:36.097374    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0ce02a_30a6_4e27_a1f9_8f8832a21269.slice/crio-58fff937cf29c724b0f285cd246813c13cac8554cf81c853f932a47c15260b52 WatchSource:0}: Error finding container 58fff937cf29c724b0f285cd246813c13cac8554cf81c853f932a47c15260b52: Status 404 returned error can't find the container with id 58fff937cf29c724b0f285cd246813c13cac8554cf81c853f932a47c15260b52
Sep 18 19:58:36.863140 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:36.863108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" event={"ID":"8f0ce02a-30a6-4e27-a1f9-8f8832a21269","Type":"ContainerStarted","Data":"c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9"}
Sep 18 19:58:36.863538 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:36.863157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" event={"ID":"8f0ce02a-30a6-4e27-a1f9-8f8832a21269","Type":"ContainerStarted","Data":"58fff937cf29c724b0f285cd246813c13cac8554cf81c853f932a47c15260b52"}
Sep 18 19:58:36.881203 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:36.881159    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" podStartSLOduration=1.881143749 podStartE2EDuration="1.881143749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:36.880107695 +0000 UTC m=+1211.134314305" watchObservedRunningTime="2026-09-18 19:58:36.881143749 +0000 UTC m=+1211.135350353"
Sep 18 19:58:37.822528 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:37.822493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:45.822127 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:45.822087    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:45.834536 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:45.834512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:58:45.908815 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:58:45.908788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:59:14.832841 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:14.832804    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"]
Sep 18 19:59:14.841521 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:59:14.841473    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:59:14.841521 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:59:14.841521 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 19:59:14.841521 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9" cmd=["/bin/sleep","15"]
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:14.841537    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" message="" err=<
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:59:14.841560    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" podUID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" containerName="main" containerID="cri-o://c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9"
Sep 18 19:59:14.841793 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:14.841606    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" podUID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" containerName="main" containerID="cri-o://c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9" gracePeriod=60
Sep 18 19:59:14.899188 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:14.899157    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" podUID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" containerName="main" probeResult="failure" output="Get \"https://10.134.0.24:8000/health\": dial tcp 10.134.0.24:8000: connect: connection refused"
Sep 18 19:59:15.174013 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.173990    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:59:15.192009 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.191988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-model-cache\") pod \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") "
Sep 18 19:59:15.192119 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192066    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-dshm\") pod \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") "
Sep 18 19:59:15.192119 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-home\") pod \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") "
Sep 18 19:59:15.192256 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192157    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j65b\" (UniqueName: \"kubernetes.io/projected/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-kube-api-access-7j65b\") pod \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") "
Sep 18 19:59:15.192256 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tls-certs\") pod \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") "
Sep 18 19:59:15.192256 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tmp-dir\") pod \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\" (UID: \"8f0ce02a-30a6-4e27-a1f9-8f8832a21269\") "
Sep 18 19:59:15.192432 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f0ce02a-30a6-4e27-a1f9-8f8832a21269" (UID: "8f0ce02a-30a6-4e27-a1f9-8f8832a21269"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:15.192489 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192462    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.192627 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192612    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f0ce02a-30a6-4e27-a1f9-8f8832a21269" (UID: "8f0ce02a-30a6-4e27-a1f9-8f8832a21269"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:15.192970 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.192946    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-home" (OuterVolumeSpecName: "home") pod "8f0ce02a-30a6-4e27-a1f9-8f8832a21269" (UID: "8f0ce02a-30a6-4e27-a1f9-8f8832a21269"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:15.194790 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.194754    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-dshm" (OuterVolumeSpecName: "dshm") pod "8f0ce02a-30a6-4e27-a1f9-8f8832a21269" (UID: "8f0ce02a-30a6-4e27-a1f9-8f8832a21269"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:15.194877 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.194854    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f0ce02a-30a6-4e27-a1f9-8f8832a21269" (UID: "8f0ce02a-30a6-4e27-a1f9-8f8832a21269"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:15.195050 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.195025    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-kube-api-access-7j65b" (OuterVolumeSpecName: "kube-api-access-7j65b") pod "8f0ce02a-30a6-4e27-a1f9-8f8832a21269" (UID: "8f0ce02a-30a6-4e27-a1f9-8f8832a21269"). InnerVolumeSpecName "kube-api-access-7j65b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:15.293453 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.293426    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.293453 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.293448    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.293453 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.293458    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j65b\" (UniqueName: \"kubernetes.io/projected/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-kube-api-access-7j65b\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.293639 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.293468    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.293639 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:15.293477    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0ce02a-30a6-4e27-a1f9-8f8832a21269-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 19:59:16.022799 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.022765    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" containerID="c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9" exitCode=0
Sep 18 19:59:16.023159 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.022806    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"
Sep 18 19:59:16.023159 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.022840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" event={"ID":"8f0ce02a-30a6-4e27-a1f9-8f8832a21269","Type":"ContainerDied","Data":"c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9"}
Sep 18 19:59:16.023159 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.022897    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np" event={"ID":"8f0ce02a-30a6-4e27-a1f9-8f8832a21269","Type":"ContainerDied","Data":"58fff937cf29c724b0f285cd246813c13cac8554cf81c853f932a47c15260b52"}
Sep 18 19:59:16.023159 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.022920    2583 scope.go:117] "RemoveContainer" containerID="c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9"
Sep 18 19:59:16.031181 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.031164    2583 scope.go:117] "RemoveContainer" containerID="c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9"
Sep 18 19:59:16.031472 ip-10-0-133-25 kubenswrapper[2583]: E0918 19:59:16.031451    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9\": container with ID starting with c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9 not found: ID does not exist" containerID="c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9"
Sep 18 19:59:16.031540 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.031481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9"} err="failed to get container status \"c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9\": rpc error: code = NotFound desc = could not find container \"c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9\": container with ID starting with c79772feeb7ca2b3b227a0aea11b47af47319b9afc0df91c1ce24669e3bf89b9 not found: ID does not exist"
Sep 18 19:59:16.037964 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.037930    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"]
Sep 18 19:59:16.039188 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:16.039165    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcx5np"]
Sep 18 19:59:17.965615 ip-10-0-133-25 kubenswrapper[2583]: I0918 19:59:17.965587    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" path="/var/lib/kubelet/pods/8f0ce02a-30a6-4e27-a1f9-8f8832a21269/volumes"
Sep 18 20:00:16.504325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.504295    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"]
Sep 18 20:00:16.504806 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.504638    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" containerName="main"
Sep 18 20:00:16.504806 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.504651    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" containerName="main"
Sep 18 20:00:16.504806 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.504709    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0ce02a-30a6-4e27-a1f9-8f8832a21269" containerName="main"
Sep 18 20:00:16.507564 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.507545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.509980 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.509935    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-n42ll\""
Sep 18 20:00:16.510201 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.510182    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 20:00:16.510201 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.510193    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 20:00:16.510372 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.510263    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 20:00:16.515909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.515888    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"]
Sep 18 20:00:16.633625 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.633589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.633625 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.633629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.633807 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.633646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgwq\" (UniqueName: \"kubernetes.io/projected/847394ab-33cd-4866-858d-fcc0d088e7c0-kube-api-access-qhgwq\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.633807 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.633679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.633807 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.633715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.633807 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.633741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847394ab-33cd-4866-858d-fcc0d088e7c0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847394ab-33cd-4866-858d-fcc0d088e7c0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734439 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734439 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734439 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgwq\" (UniqueName: \"kubernetes.io/projected/847394ab-33cd-4866-858d-fcc0d088e7c0-kube-api-access-qhgwq\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734551 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734604 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734812 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.734865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.734830    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.736692 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.736675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.736961 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.736945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847394ab-33cd-4866-858d-fcc0d088e7c0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.744147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.744120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgwq\" (UniqueName: \"kubernetes.io/projected/847394ab-33cd-4866-858d-fcc0d088e7c0-kube-api-access-qhgwq\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:16.817724 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:16.817689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:17.090839 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:17.090816    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"]
Sep 18 20:00:17.093015 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:00:17.092988    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847394ab_33cd_4866_858d_fcc0d088e7c0.slice/crio-0c165429fbd2d945f4039b7cd44ce41f3cd0bb250d11b0beb6e47e9e2ae06407 WatchSource:0}: Error finding container 0c165429fbd2d945f4039b7cd44ce41f3cd0bb250d11b0beb6e47e9e2ae06407: Status 404 returned error can't find the container with id 0c165429fbd2d945f4039b7cd44ce41f3cd0bb250d11b0beb6e47e9e2ae06407
Sep 18 20:00:17.266647 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:17.266576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" event={"ID":"847394ab-33cd-4866-858d-fcc0d088e7c0","Type":"ContainerStarted","Data":"b566198848537cbbe59c71b1a49ada32165c938a6749e61cddaa94f0e7aa1a54"}
Sep 18 20:00:17.266831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:17.266664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" event={"ID":"847394ab-33cd-4866-858d-fcc0d088e7c0","Type":"ContainerStarted","Data":"0c165429fbd2d945f4039b7cd44ce41f3cd0bb250d11b0beb6e47e9e2ae06407"}
Sep 18 20:00:17.294455 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:17.294406    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" podStartSLOduration=1.294389026 podStartE2EDuration="1.294389026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:17.292960573 +0000 UTC m=+1311.547167179" watchObservedRunningTime="2026-09-18 20:00:17.294389026 +0000 UTC m=+1311.548595630"
Sep 18 20:00:17.818600 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:17.818572    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:26.818440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:26.818407    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:26.830790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:26.830768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:27.316769 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:27.316738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:49.938624 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:49.938533    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"]
Sep 18 20:00:49.949525 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:00:49.949482    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:00:49.949525 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:00:49.949525 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:00:49.949525 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="b566198848537cbbe59c71b1a49ada32165c938a6749e61cddaa94f0e7aa1a54" cmd=["/bin/sleep","15"]
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:49.949538    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" message="" err=<
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:00:49.949550    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" podUID="847394ab-33cd-4866-858d-fcc0d088e7c0" containerName="main" containerID="cri-o://b566198848537cbbe59c71b1a49ada32165c938a6749e61cddaa94f0e7aa1a54"
Sep 18 20:00:49.949780 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:49.949598    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" podUID="847394ab-33cd-4866-858d-fcc0d088e7c0" containerName="main" containerID="cri-o://b566198848537cbbe59c71b1a49ada32165c938a6749e61cddaa94f0e7aa1a54" gracePeriod=60
Sep 18 20:00:50.306716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.306682    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" podUID="847394ab-33cd-4866-858d-fcc0d088e7c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Sep 18 20:00:50.398064 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.398026    2583 generic.go:356] "Generic (PLEG): container finished" podID="847394ab-33cd-4866-858d-fcc0d088e7c0" containerID="b566198848537cbbe59c71b1a49ada32165c938a6749e61cddaa94f0e7aa1a54" exitCode=0
Sep 18 20:00:50.398212 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.398098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" event={"ID":"847394ab-33cd-4866-858d-fcc0d088e7c0","Type":"ContainerDied","Data":"b566198848537cbbe59c71b1a49ada32165c938a6749e61cddaa94f0e7aa1a54"}
Sep 18 20:00:50.693008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.692988    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:50.785124 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-dshm\") pod \"847394ab-33cd-4866-858d-fcc0d088e7c0\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") "
Sep 18 20:00:50.785124 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785130    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847394ab-33cd-4866-858d-fcc0d088e7c0-tls-certs\") pod \"847394ab-33cd-4866-858d-fcc0d088e7c0\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") "
Sep 18 20:00:50.785333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhgwq\" (UniqueName: \"kubernetes.io/projected/847394ab-33cd-4866-858d-fcc0d088e7c0-kube-api-access-qhgwq\") pod \"847394ab-33cd-4866-858d-fcc0d088e7c0\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") "
Sep 18 20:00:50.785333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-tmp-dir\") pod \"847394ab-33cd-4866-858d-fcc0d088e7c0\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") "
Sep 18 20:00:50.785333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-model-cache\") pod \"847394ab-33cd-4866-858d-fcc0d088e7c0\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") "
Sep 18 20:00:50.785333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785243    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-home\") pod \"847394ab-33cd-4866-858d-fcc0d088e7c0\" (UID: \"847394ab-33cd-4866-858d-fcc0d088e7c0\") "
Sep 18 20:00:50.785586 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "847394ab-33cd-4866-858d-fcc0d088e7c0" (UID: "847394ab-33cd-4866-858d-fcc0d088e7c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:50.785586 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "847394ab-33cd-4866-858d-fcc0d088e7c0" (UID: "847394ab-33cd-4866-858d-fcc0d088e7c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:50.785745 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.785656    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-home" (OuterVolumeSpecName: "home") pod "847394ab-33cd-4866-858d-fcc0d088e7c0" (UID: "847394ab-33cd-4866-858d-fcc0d088e7c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:50.787492 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.787470    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847394ab-33cd-4866-858d-fcc0d088e7c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "847394ab-33cd-4866-858d-fcc0d088e7c0" (UID: "847394ab-33cd-4866-858d-fcc0d088e7c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:50.787919 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.787893    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-dshm" (OuterVolumeSpecName: "dshm") pod "847394ab-33cd-4866-858d-fcc0d088e7c0" (UID: "847394ab-33cd-4866-858d-fcc0d088e7c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:50.787919 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.787903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847394ab-33cd-4866-858d-fcc0d088e7c0-kube-api-access-qhgwq" (OuterVolumeSpecName: "kube-api-access-qhgwq") pod "847394ab-33cd-4866-858d-fcc0d088e7c0" (UID: "847394ab-33cd-4866-858d-fcc0d088e7c0"). InnerVolumeSpecName "kube-api-access-qhgwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:50.885953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.885889    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:00:50.885953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.885910    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:00:50.885953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.885919    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847394ab-33cd-4866-858d-fcc0d088e7c0-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:00:50.885953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.885928    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgwq\" (UniqueName: \"kubernetes.io/projected/847394ab-33cd-4866-858d-fcc0d088e7c0-kube-api-access-qhgwq\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:00:50.885953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.885937    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:00:50.885953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:50.885944    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847394ab-33cd-4866-858d-fcc0d088e7c0-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:00:51.402986 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:51.402946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2" event={"ID":"847394ab-33cd-4866-858d-fcc0d088e7c0","Type":"ContainerDied","Data":"0c165429fbd2d945f4039b7cd44ce41f3cd0bb250d11b0beb6e47e9e2ae06407"}
Sep 18 20:00:51.403460 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:51.403012    2583 scope.go:117] "RemoveContainer" containerID="b566198848537cbbe59c71b1a49ada32165c938a6749e61cddaa94f0e7aa1a54"
Sep 18 20:00:51.403460 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:51.402961    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"
Sep 18 20:00:51.421986 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:51.421963    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"]
Sep 18 20:00:51.425945 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:51.425922    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-lbcp2"]
Sep 18 20:00:51.967201 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:00:51.967169    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847394ab-33cd-4866-858d-fcc0d088e7c0" path="/var/lib/kubelet/pods/847394ab-33cd-4866-858d-fcc0d088e7c0/volumes"
Sep 18 20:01:52.708719 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.708686    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"]
Sep 18 20:01:52.709264 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.709089    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847394ab-33cd-4866-858d-fcc0d088e7c0" containerName="main"
Sep 18 20:01:52.709264 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.709107    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="847394ab-33cd-4866-858d-fcc0d088e7c0" containerName="main"
Sep 18 20:01:52.709264 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.709168    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="847394ab-33cd-4866-858d-fcc0d088e7c0" containerName="main"
Sep 18 20:01:52.711901 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.711880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.715240 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.715218    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 20:01:52.715340 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.715219    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 20:01:52.715340 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.715321    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 20:01:52.715475 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.715340    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-pwlm2\""
Sep 18 20:01:52.723089 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.723066    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"]
Sep 18 20:01:52.836293 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.836266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frksc\" (UniqueName: \"kubernetes.io/projected/37399457-fdec-4fc0-8463-29644b34e3f8-kube-api-access-frksc\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.836413 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.836307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.836413 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.836382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.836491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.836427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.836491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.836466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.836491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.836481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37399457-fdec-4fc0-8463-29644b34e3f8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frksc\" (UniqueName: \"kubernetes.io/projected/37399457-fdec-4fc0-8463-29644b34e3f8-kube-api-access-frksc\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937522 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937522 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937522 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937522 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937522 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37399457-fdec-4fc0-8463-29644b34e3f8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937917 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.937917 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.937890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.939970 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.939947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.940103 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.940087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37399457-fdec-4fc0-8463-29644b34e3f8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:52.951580 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:52.951556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frksc\" (UniqueName: \"kubernetes.io/projected/37399457-fdec-4fc0-8463-29644b34e3f8-kube-api-access-frksc\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:53.022060 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:53.022028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:01:53.296340 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:53.296271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"]
Sep 18 20:01:53.299183 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:01:53.299156    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37399457_fdec_4fc0_8463_29644b34e3f8.slice/crio-6320805f1cdfdc75afa7d9f8f3d3af3ea9c7f07021332b47be526bf87141f3cd WatchSource:0}: Error finding container 6320805f1cdfdc75afa7d9f8f3d3af3ea9c7f07021332b47be526bf87141f3cd: Status 404 returned error can't find the container with id 6320805f1cdfdc75afa7d9f8f3d3af3ea9c7f07021332b47be526bf87141f3cd
Sep 18 20:01:53.652797 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:53.652712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" event={"ID":"37399457-fdec-4fc0-8463-29644b34e3f8","Type":"ContainerStarted","Data":"71cb83e10636ffa40ede65ca022b56bccfa6cb4cba8494e3a7fc8f0eb492956a"}
Sep 18 20:01:53.652797 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:53.652762    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" event={"ID":"37399457-fdec-4fc0-8463-29644b34e3f8","Type":"ContainerStarted","Data":"6320805f1cdfdc75afa7d9f8f3d3af3ea9c7f07021332b47be526bf87141f3cd"}
Sep 18 20:01:53.695333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:53.695276    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" podStartSLOduration=1.695262668 podStartE2EDuration="1.695262668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:53.69040314 +0000 UTC m=+1407.944609740" watchObservedRunningTime="2026-09-18 20:01:53.695262668 +0000 UTC m=+1407.949469305"
Sep 18 20:01:54.022859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:01:54.022830    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:02:03.022516 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:03.022474    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:02:03.034921 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:03.034889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:02:03.709199 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:03.709167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:02:28.548952 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:28.548918    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"]
Sep 18 20:02:28.557957 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:02:28.557924    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:02:28.557957 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:02:28.557957 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:02:28.557957 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="71cb83e10636ffa40ede65ca022b56bccfa6cb4cba8494e3a7fc8f0eb492956a" cmd=["/bin/sleep","15"]
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:28.557975    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" message="" err=<
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:02:28.557987    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" podUID="37399457-fdec-4fc0-8463-29644b34e3f8" containerName="main" containerID="cri-o://71cb83e10636ffa40ede65ca022b56bccfa6cb4cba8494e3a7fc8f0eb492956a"
Sep 18 20:02:28.558187 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:28.558047    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" podUID="37399457-fdec-4fc0-8463-29644b34e3f8" containerName="main" containerID="cri-o://71cb83e10636ffa40ede65ca022b56bccfa6cb4cba8494e3a7fc8f0eb492956a" gracePeriod=60
Sep 18 20:02:28.698405 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:28.698375    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" podUID="37399457-fdec-4fc0-8463-29644b34e3f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Sep 18 20:02:28.803933 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:28.803856    2583 generic.go:356] "Generic (PLEG): container finished" podID="37399457-fdec-4fc0-8463-29644b34e3f8" containerID="71cb83e10636ffa40ede65ca022b56bccfa6cb4cba8494e3a7fc8f0eb492956a" exitCode=0
Sep 18 20:02:28.803933 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:28.803923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" event={"ID":"37399457-fdec-4fc0-8463-29644b34e3f8","Type":"ContainerDied","Data":"71cb83e10636ffa40ede65ca022b56bccfa6cb4cba8494e3a7fc8f0eb492956a"}
Sep 18 20:02:28.898812 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:28.898790    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:02:29.018788 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.018759    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-model-cache\") pod \"37399457-fdec-4fc0-8463-29644b34e3f8\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") "
Sep 18 20:02:29.018955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.018802    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37399457-fdec-4fc0-8463-29644b34e3f8-tls-certs\") pod \"37399457-fdec-4fc0-8463-29644b34e3f8\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") "
Sep 18 20:02:29.018955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.018867    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-tmp-dir\") pod \"37399457-fdec-4fc0-8463-29644b34e3f8\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") "
Sep 18 20:02:29.018955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.018885    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-dshm\") pod \"37399457-fdec-4fc0-8463-29644b34e3f8\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") "
Sep 18 20:02:29.018955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.018923    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frksc\" (UniqueName: \"kubernetes.io/projected/37399457-fdec-4fc0-8463-29644b34e3f8-kube-api-access-frksc\") pod \"37399457-fdec-4fc0-8463-29644b34e3f8\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") "
Sep 18 20:02:29.018955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.018941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-home\") pod \"37399457-fdec-4fc0-8463-29644b34e3f8\" (UID: \"37399457-fdec-4fc0-8463-29644b34e3f8\") "
Sep 18 20:02:29.019221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.019053    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "37399457-fdec-4fc0-8463-29644b34e3f8" (UID: "37399457-fdec-4fc0-8463-29644b34e3f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.019221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.019175    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37399457-fdec-4fc0-8463-29644b34e3f8" (UID: "37399457-fdec-4fc0-8463-29644b34e3f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.019221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.019198    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-home" (OuterVolumeSpecName: "home") pod "37399457-fdec-4fc0-8463-29644b34e3f8" (UID: "37399457-fdec-4fc0-8463-29644b34e3f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.021258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.021239    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-dshm" (OuterVolumeSpecName: "dshm") pod "37399457-fdec-4fc0-8463-29644b34e3f8" (UID: "37399457-fdec-4fc0-8463-29644b34e3f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:29.021350 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.021332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37399457-fdec-4fc0-8463-29644b34e3f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37399457-fdec-4fc0-8463-29644b34e3f8" (UID: "37399457-fdec-4fc0-8463-29644b34e3f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:29.021350 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.021335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37399457-fdec-4fc0-8463-29644b34e3f8-kube-api-access-frksc" (OuterVolumeSpecName: "kube-api-access-frksc") pod "37399457-fdec-4fc0-8463-29644b34e3f8" (UID: "37399457-fdec-4fc0-8463-29644b34e3f8"). InnerVolumeSpecName "kube-api-access-frksc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:29.119904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.119830    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.119904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.119864    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.119904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.119874    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frksc\" (UniqueName: \"kubernetes.io/projected/37399457-fdec-4fc0-8463-29644b34e3f8-kube-api-access-frksc\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.119904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.119883    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.119904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.119892    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37399457-fdec-4fc0-8463-29644b34e3f8-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.119904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.119900    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37399457-fdec-4fc0-8463-29644b34e3f8-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:02:29.808973 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.808942    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"
Sep 18 20:02:29.808973 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.808936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv" event={"ID":"37399457-fdec-4fc0-8463-29644b34e3f8","Type":"ContainerDied","Data":"6320805f1cdfdc75afa7d9f8f3d3af3ea9c7f07021332b47be526bf87141f3cd"}
Sep 18 20:02:29.809572 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.809028    2583 scope.go:117] "RemoveContainer" containerID="71cb83e10636ffa40ede65ca022b56bccfa6cb4cba8494e3a7fc8f0eb492956a"
Sep 18 20:02:29.829348 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.829324    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"]
Sep 18 20:02:29.830418 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.830399    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bwgxv"]
Sep 18 20:02:29.965802 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:02:29.965773    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37399457-fdec-4fc0-8463-29644b34e3f8" path="/var/lib/kubelet/pods/37399457-fdec-4fc0-8463-29644b34e3f8/volumes"
Sep 18 20:03:23.888737 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.888706    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"]
Sep 18 20:03:23.889180 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.889111    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37399457-fdec-4fc0-8463-29644b34e3f8" containerName="main"
Sep 18 20:03:23.889180 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.889125    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="37399457-fdec-4fc0-8463-29644b34e3f8" containerName="main"
Sep 18 20:03:23.889180 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.889176    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="37399457-fdec-4fc0-8463-29644b34e3f8" containerName="main"
Sep 18 20:03:23.892066 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.892047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:23.895488 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.895467    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 20:03:23.895488 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.895474    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-wgvnw\""
Sep 18 20:03:23.895670 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.895473    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 20:03:23.895670 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.895559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 20:03:23.901046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.901025    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"]
Sep 18 20:03:23.912083 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.912060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:23.912167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.912094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:23.912167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.912115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgnh9\" (UniqueName: \"kubernetes.io/projected/59a19fca-c316-4ab2-ad7d-67eed73c6b26-kube-api-access-pgnh9\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:23.912167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.912159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:23.912272 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.912191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:23.912272 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:23.912219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.012888 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.012859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013047 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.012898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgnh9\" (UniqueName: \"kubernetes.io/projected/59a19fca-c316-4ab2-ad7d-67eed73c6b26-kube-api-access-pgnh9\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013047 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.012969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013047 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.013035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.013064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.013118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013345 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.013326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013441 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.013417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.013498 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.013441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.015477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.015456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.015823 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.015804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.020592 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.020574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgnh9\" (UniqueName: \"kubernetes.io/projected/59a19fca-c316-4ab2-ad7d-67eed73c6b26-kube-api-access-pgnh9\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-z98jv\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.202984 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.202902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:24.470739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:24.470712    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"]
Sep 18 20:03:24.472682 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:03:24.472657    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a19fca_c316_4ab2_ad7d_67eed73c6b26.slice/crio-335b7fe36240973f91fe8455c547cfa500f763b3e5731182da1af2e86085923d WatchSource:0}: Error finding container 335b7fe36240973f91fe8455c547cfa500f763b3e5731182da1af2e86085923d: Status 404 returned error can't find the container with id 335b7fe36240973f91fe8455c547cfa500f763b3e5731182da1af2e86085923d
Sep 18 20:03:25.024931 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:25.024892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" event={"ID":"59a19fca-c316-4ab2-ad7d-67eed73c6b26","Type":"ContainerStarted","Data":"40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7"}
Sep 18 20:03:25.025279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:25.024973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" event={"ID":"59a19fca-c316-4ab2-ad7d-67eed73c6b26","Type":"ContainerStarted","Data":"335b7fe36240973f91fe8455c547cfa500f763b3e5731182da1af2e86085923d"}
Sep 18 20:03:25.041717 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:25.041668    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" podStartSLOduration=2.041649376 podStartE2EDuration="2.041649376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:25.040745281 +0000 UTC m=+1499.294951884" watchObservedRunningTime="2026-09-18 20:03:25.041649376 +0000 UTC m=+1499.295855984"
Sep 18 20:03:25.203623 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:25.203588    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:27.948909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:27.948886    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:03:27.953095 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:27.953075    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:03:34.204047 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:34.204004    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:34.217238 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:34.217205    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:35.081227 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:35.081201    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:58.139491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.139413    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"]
Sep 18 20:03:58.149851 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:03:58.149808    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:03:58.149851 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:03:58.149851 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:03:58.149851 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7" cmd=["/bin/sleep","15"]
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.149871    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" message="" err=<
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:03:58.149890    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" podUID="59a19fca-c316-4ab2-ad7d-67eed73c6b26" containerName="main" containerID="cri-o://40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7"
Sep 18 20:03:58.150157 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.149935    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" podUID="59a19fca-c316-4ab2-ad7d-67eed73c6b26" containerName="main" containerID="cri-o://40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7" gracePeriod=60
Sep 18 20:03:58.484639 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.484616    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:58.575552 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575522    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tmp-dir\") pod \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") "
Sep 18 20:03:58.575717 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-dshm\") pod \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") "
Sep 18 20:03:58.575717 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgnh9\" (UniqueName: \"kubernetes.io/projected/59a19fca-c316-4ab2-ad7d-67eed73c6b26-kube-api-access-pgnh9\") pod \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") "
Sep 18 20:03:58.575717 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-model-cache\") pod \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") "
Sep 18 20:03:58.575717 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575704    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-home\") pod \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") "
Sep 18 20:03:58.575938 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575823    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59a19fca-c316-4ab2-ad7d-67eed73c6b26" (UID: "59a19fca-c316-4ab2-ad7d-67eed73c6b26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.575938 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575899    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tls-certs\") pod \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\" (UID: \"59a19fca-c316-4ab2-ad7d-67eed73c6b26\") "
Sep 18 20:03:58.575938 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.575913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-home" (OuterVolumeSpecName: "home") pod "59a19fca-c316-4ab2-ad7d-67eed73c6b26" (UID: "59a19fca-c316-4ab2-ad7d-67eed73c6b26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.576077 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.576004    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-model-cache" (OuterVolumeSpecName: "model-cache") pod "59a19fca-c316-4ab2-ad7d-67eed73c6b26" (UID: "59a19fca-c316-4ab2-ad7d-67eed73c6b26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.576157 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.576141    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.576206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.576163    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.576206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.576176    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.578088 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.578059    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59a19fca-c316-4ab2-ad7d-67eed73c6b26" (UID: "59a19fca-c316-4ab2-ad7d-67eed73c6b26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:58.578206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.578121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a19fca-c316-4ab2-ad7d-67eed73c6b26-kube-api-access-pgnh9" (OuterVolumeSpecName: "kube-api-access-pgnh9") pod "59a19fca-c316-4ab2-ad7d-67eed73c6b26" (UID: "59a19fca-c316-4ab2-ad7d-67eed73c6b26"). InnerVolumeSpecName "kube-api-access-pgnh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:58.578206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.578148    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-dshm" (OuterVolumeSpecName: "dshm") pod "59a19fca-c316-4ab2-ad7d-67eed73c6b26" (UID: "59a19fca-c316-4ab2-ad7d-67eed73c6b26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.676674 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.676616    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a19fca-c316-4ab2-ad7d-67eed73c6b26-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.676674 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.676636    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgnh9\" (UniqueName: \"kubernetes.io/projected/59a19fca-c316-4ab2-ad7d-67eed73c6b26-kube-api-access-pgnh9\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:03:58.676674 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:58.676646    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a19fca-c316-4ab2-ad7d-67eed73c6b26-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:03:59.165115 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.165079    2583 generic.go:356] "Generic (PLEG): container finished" podID="59a19fca-c316-4ab2-ad7d-67eed73c6b26" containerID="40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7" exitCode=0
Sep 18 20:03:59.165115 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.165118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" event={"ID":"59a19fca-c316-4ab2-ad7d-67eed73c6b26","Type":"ContainerDied","Data":"40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7"}
Sep 18 20:03:59.165631 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.165134    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"
Sep 18 20:03:59.165631 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.165160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv" event={"ID":"59a19fca-c316-4ab2-ad7d-67eed73c6b26","Type":"ContainerDied","Data":"335b7fe36240973f91fe8455c547cfa500f763b3e5731182da1af2e86085923d"}
Sep 18 20:03:59.165631 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.165180    2583 scope.go:117] "RemoveContainer" containerID="40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7"
Sep 18 20:03:59.177406 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.177380    2583 scope.go:117] "RemoveContainer" containerID="40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7"
Sep 18 20:03:59.177640 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:03:59.177621    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7\": container with ID starting with 40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7 not found: ID does not exist" containerID="40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7"
Sep 18 20:03:59.177716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.177651    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7"} err="failed to get container status \"40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7\": rpc error: code = NotFound desc = could not find container \"40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7\": container with ID starting with 40d7a0937fcf4da539db155708636d3bd3dbf930c14e2434843c0b17b3d6afd7 not found: ID does not exist"
Sep 18 20:03:59.186820 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.186800    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"]
Sep 18 20:03:59.190601 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.190582    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-z98jv"]
Sep 18 20:03:59.966469 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:03:59.966435    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a19fca-c316-4ab2-ad7d-67eed73c6b26" path="/var/lib/kubelet/pods/59a19fca-c316-4ab2-ad7d-67eed73c6b26/volumes"
Sep 18 20:04:51.340162 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.340131    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"]
Sep 18 20:04:51.340646 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.340576    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a19fca-c316-4ab2-ad7d-67eed73c6b26" containerName="main"
Sep 18 20:04:51.340646 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.340593    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a19fca-c316-4ab2-ad7d-67eed73c6b26" containerName="main"
Sep 18 20:04:51.340734 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.340721    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a19fca-c316-4ab2-ad7d-67eed73c6b26" containerName="main"
Sep 18 20:04:51.343609 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.343591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.346066 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.346046    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 20:04:51.347122 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.347099    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 20:04:51.347231 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.347134    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 20:04:51.347298 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.347235    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-ttxps\""
Sep 18 20:04:51.354884 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.354864    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"]
Sep 18 20:04:51.443120 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.443091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e1a68-7fa5-45f4-8610-697ad84b5e48-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.443120 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.443123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.443310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.443151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92cm5\" (UniqueName: \"kubernetes.io/projected/635e1a68-7fa5-45f4-8610-697ad84b5e48-kube-api-access-92cm5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.443310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.443203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.443310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.443219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.443310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.443295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.540684 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.540657    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"]
Sep 18 20:04:51.543772 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.543752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.544010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.543989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e1a68-7fa5-45f4-8610-697ad84b5e48-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544220 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92cm5\" (UniqueName: \"kubernetes.io/projected/635e1a68-7fa5-45f4-8610-697ad84b5e48-kube-api-access-92cm5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544220 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544220 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544541 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544713 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.544949 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.544910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.546388 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.546339    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-gq5lt\""
Sep 18 20:04:51.547012 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.546987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.547093 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.547051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e1a68-7fa5-45f4-8610-697ad84b5e48-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.555134 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.555110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92cm5\" (UniqueName: \"kubernetes.io/projected/635e1a68-7fa5-45f4-8610-697ad84b5e48-kube-api-access-92cm5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.555927 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.555909    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"]
Sep 18 20:04:51.645047 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.644989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lznm\" (UniqueName: \"kubernetes.io/projected/e6671223-e56e-41c8-bba0-bc0711a1cfb7-kube-api-access-5lznm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.645047 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.645036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6671223-e56e-41c8-bba0-bc0711a1cfb7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.654022 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.653999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:51.746115 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.746083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lznm\" (UniqueName: \"kubernetes.io/projected/e6671223-e56e-41c8-bba0-bc0711a1cfb7-kube-api-access-5lznm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.746280 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.746135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6671223-e56e-41c8-bba0-bc0711a1cfb7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.748500 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.748482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6671223-e56e-41c8-bba0-bc0711a1cfb7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.754836 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.754820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lznm\" (UniqueName: \"kubernetes.io/projected/e6671223-e56e-41c8-bba0-bc0711a1cfb7-kube-api-access-5lznm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.871772 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.871736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:51.922623 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:51.922421    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"]
Sep 18 20:04:51.925141 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:04:51.924911    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635e1a68_7fa5_45f4_8610_697ad84b5e48.slice/crio-44d3a9fcd7972201b3079ea86bb75b2a29ea44dcb4413523dd958f32c04cf561 WatchSource:0}: Error finding container 44d3a9fcd7972201b3079ea86bb75b2a29ea44dcb4413523dd958f32c04cf561: Status 404 returned error can't find the container with id 44d3a9fcd7972201b3079ea86bb75b2a29ea44dcb4413523dd958f32c04cf561
Sep 18 20:04:52.140280 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:52.140249    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"]
Sep 18 20:04:52.143284 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:04:52.143253    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6671223_e56e_41c8_bba0_bc0711a1cfb7.slice/crio-03e05bea85f7e3410649176efeb94fe5cfcc35b6c94f8cec24c33c7b3d723410 WatchSource:0}: Error finding container 03e05bea85f7e3410649176efeb94fe5cfcc35b6c94f8cec24c33c7b3d723410: Status 404 returned error can't find the container with id 03e05bea85f7e3410649176efeb94fe5cfcc35b6c94f8cec24c33c7b3d723410
Sep 18 20:04:52.145349 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:52.145331    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:04:52.379043 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:52.379004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" event={"ID":"e6671223-e56e-41c8-bba0-bc0711a1cfb7","Type":"ContainerStarted","Data":"03e05bea85f7e3410649176efeb94fe5cfcc35b6c94f8cec24c33c7b3d723410"}
Sep 18 20:04:52.380594 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:52.380567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" event={"ID":"635e1a68-7fa5-45f4-8610-697ad84b5e48","Type":"ContainerStarted","Data":"f71aeea813dacb9f577e57272f01652343e282a7c6f39c30f99d35499db1c8aa"}
Sep 18 20:04:52.380704 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:52.380610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" event={"ID":"635e1a68-7fa5-45f4-8610-697ad84b5e48","Type":"ContainerStarted","Data":"44d3a9fcd7972201b3079ea86bb75b2a29ea44dcb4413523dd958f32c04cf561"}
Sep 18 20:04:52.398629 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:52.398579    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" podStartSLOduration=1.398564968 podStartE2EDuration="1.398564968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:52.397165067 +0000 UTC m=+1586.651371684" watchObservedRunningTime="2026-09-18 20:04:52.398564968 +0000 UTC m=+1586.652771571"
Sep 18 20:04:52.654735 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:52.654664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:04:55.397160 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:55.397127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" event={"ID":"e6671223-e56e-41c8-bba0-bc0711a1cfb7","Type":"ContainerStarted","Data":"fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac"}
Sep 18 20:04:55.397554 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:55.397231    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:04:55.412148 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:04:55.412103    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" podStartSLOduration=2.114826632 podStartE2EDuration="4.412090292s" totalImagesPullingTime="2.29726366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:55.410835661 +0000 UTC m=+1589.665042267" watchObservedRunningTime="2026-09-18 20:04:55.412090292 +0000 UTC m=+1589.666296918"
Sep 18 20:05:01.654923 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:05:01.654871    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:05:01.666947 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:05:01.666920    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:05:02.437825 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:05:02.437798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:05:26.402985 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:05:26.402956    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:06:04.015018 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.014983    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"]
Sep 18 20:06:04.021815 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.021787    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"]
Sep 18 20:06:04.024456 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:06:04.024425    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:06:04.024456 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:06:04.024456 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:06:04.024456 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="f71aeea813dacb9f577e57272f01652343e282a7c6f39c30f99d35499db1c8aa" cmd=["/bin/sleep","15"]
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.024471    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" message="" err=<
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:06:04.024485    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" podUID="635e1a68-7fa5-45f4-8610-697ad84b5e48" containerName="main" containerID="cri-o://f71aeea813dacb9f577e57272f01652343e282a7c6f39c30f99d35499db1c8aa"
Sep 18 20:06:04.024709 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.024521    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" podUID="635e1a68-7fa5-45f4-8610-697ad84b5e48" containerName="main" containerID="cri-o://f71aeea813dacb9f577e57272f01652343e282a7c6f39c30f99d35499db1c8aa" gracePeriod=60
Sep 18 20:06:04.427687 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.427593    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" podUID="635e1a68-7fa5-45f4-8610-697ad84b5e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 18 20:06:04.677608 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.677573    2583 generic.go:356] "Generic (PLEG): container finished" podID="635e1a68-7fa5-45f4-8610-697ad84b5e48" containerID="f71aeea813dacb9f577e57272f01652343e282a7c6f39c30f99d35499db1c8aa" exitCode=0
Sep 18 20:06:04.677792 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.677644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" event={"ID":"635e1a68-7fa5-45f4-8610-697ad84b5e48","Type":"ContainerDied","Data":"f71aeea813dacb9f577e57272f01652343e282a7c6f39c30f99d35499db1c8aa"}
Sep 18 20:06:04.754178 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.754155    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:06:04.795958 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.795930    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-model-cache\") pod \"635e1a68-7fa5-45f4-8610-697ad84b5e48\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") "
Sep 18 20:06:04.796094 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.795977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-dshm\") pod \"635e1a68-7fa5-45f4-8610-697ad84b5e48\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") "
Sep 18 20:06:04.796094 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.796009    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92cm5\" (UniqueName: \"kubernetes.io/projected/635e1a68-7fa5-45f4-8610-697ad84b5e48-kube-api-access-92cm5\") pod \"635e1a68-7fa5-45f4-8610-697ad84b5e48\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") "
Sep 18 20:06:04.796094 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.796040    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-tmp-dir\") pod \"635e1a68-7fa5-45f4-8610-697ad84b5e48\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") "
Sep 18 20:06:04.796094 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.796055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-home\") pod \"635e1a68-7fa5-45f4-8610-697ad84b5e48\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") "
Sep 18 20:06:04.796094 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.796076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e1a68-7fa5-45f4-8610-697ad84b5e48-tls-certs\") pod \"635e1a68-7fa5-45f4-8610-697ad84b5e48\" (UID: \"635e1a68-7fa5-45f4-8610-697ad84b5e48\") "
Sep 18 20:06:04.796399 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.796189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-model-cache" (OuterVolumeSpecName: "model-cache") pod "635e1a68-7fa5-45f4-8610-697ad84b5e48" (UID: "635e1a68-7fa5-45f4-8610-697ad84b5e48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.796399 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.796318    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "635e1a68-7fa5-45f4-8610-697ad84b5e48" (UID: "635e1a68-7fa5-45f4-8610-697ad84b5e48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.796399 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.796330    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-home" (OuterVolumeSpecName: "home") pod "635e1a68-7fa5-45f4-8610-697ad84b5e48" (UID: "635e1a68-7fa5-45f4-8610-697ad84b5e48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.798409 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.798387    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-dshm" (OuterVolumeSpecName: "dshm") pod "635e1a68-7fa5-45f4-8610-697ad84b5e48" (UID: "635e1a68-7fa5-45f4-8610-697ad84b5e48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.798515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.798465    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635e1a68-7fa5-45f4-8610-697ad84b5e48-kube-api-access-92cm5" (OuterVolumeSpecName: "kube-api-access-92cm5") pod "635e1a68-7fa5-45f4-8610-697ad84b5e48" (UID: "635e1a68-7fa5-45f4-8610-697ad84b5e48"). InnerVolumeSpecName "kube-api-access-92cm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:04.798515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.798463    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635e1a68-7fa5-45f4-8610-697ad84b5e48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "635e1a68-7fa5-45f4-8610-697ad84b5e48" (UID: "635e1a68-7fa5-45f4-8610-697ad84b5e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:04.897051 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.897010    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.897051 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.897046    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92cm5\" (UniqueName: \"kubernetes.io/projected/635e1a68-7fa5-45f4-8610-697ad84b5e48-kube-api-access-92cm5\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.897051 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.897056    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.897253 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.897066    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.897253 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.897075    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e1a68-7fa5-45f4-8610-697ad84b5e48-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.897253 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:04.897082    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e1a68-7fa5-45f4-8610-697ad84b5e48-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:05.683779 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:05.683745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2" event={"ID":"635e1a68-7fa5-45f4-8610-697ad84b5e48","Type":"ContainerDied","Data":"44d3a9fcd7972201b3079ea86bb75b2a29ea44dcb4413523dd958f32c04cf561"}
Sep 18 20:06:05.683779 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:05.683771    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"
Sep 18 20:06:05.684236 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:05.683810    2583 scope.go:117] "RemoveContainer" containerID="f71aeea813dacb9f577e57272f01652343e282a7c6f39c30f99d35499db1c8aa"
Sep 18 20:06:05.703201 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:05.703171    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"]
Sep 18 20:06:05.708877 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:05.708856    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-hmkg2"]
Sep 18 20:06:05.966658 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:05.966587    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635e1a68-7fa5-45f4-8610-697ad84b5e48" path="/var/lib/kubelet/pods/635e1a68-7fa5-45f4-8610-697ad84b5e48/volumes"
Sep 18 20:06:06.402602 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:06.402569    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:16.402755 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:16.402718    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:19.040597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:19.040541    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main" containerID="cri-o://fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac" gracePeriod=45
Sep 18 20:06:26.403473 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:26.403431    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:26.403855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:26.403528    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:06:36.402808 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:36.402773    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:46.403122 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:46.403041    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:49.279201 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.279176    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:06:49.436197 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.436122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6671223-e56e-41c8-bba0-bc0711a1cfb7-tls-certs\") pod \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") "
Sep 18 20:06:49.436197 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.436188    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lznm\" (UniqueName: \"kubernetes.io/projected/e6671223-e56e-41c8-bba0-bc0711a1cfb7-kube-api-access-5lznm\") pod \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\" (UID: \"e6671223-e56e-41c8-bba0-bc0711a1cfb7\") "
Sep 18 20:06:49.438310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.438283    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6671223-e56e-41c8-bba0-bc0711a1cfb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6671223-e56e-41c8-bba0-bc0711a1cfb7" (UID: "e6671223-e56e-41c8-bba0-bc0711a1cfb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:49.438310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.438298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6671223-e56e-41c8-bba0-bc0711a1cfb7-kube-api-access-5lznm" (OuterVolumeSpecName: "kube-api-access-5lznm") pod "e6671223-e56e-41c8-bba0-bc0711a1cfb7" (UID: "e6671223-e56e-41c8-bba0-bc0711a1cfb7"). InnerVolumeSpecName "kube-api-access-5lznm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:49.536855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.536807    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lznm\" (UniqueName: \"kubernetes.io/projected/e6671223-e56e-41c8-bba0-bc0711a1cfb7-kube-api-access-5lznm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:49.536855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.536852    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6671223-e56e-41c8-bba0-bc0711a1cfb7-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:06:49.864620 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.864584    2583 generic.go:356] "Generic (PLEG): container finished" podID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerID="fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac" exitCode=0
Sep 18 20:06:49.864801 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.864636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" event={"ID":"e6671223-e56e-41c8-bba0-bc0711a1cfb7","Type":"ContainerDied","Data":"fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac"}
Sep 18 20:06:49.864801 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.864638    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"
Sep 18 20:06:49.864801 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.864704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p" event={"ID":"e6671223-e56e-41c8-bba0-bc0711a1cfb7","Type":"ContainerDied","Data":"03e05bea85f7e3410649176efeb94fe5cfcc35b6c94f8cec24c33c7b3d723410"}
Sep 18 20:06:49.864801 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.864726    2583 scope.go:117] "RemoveContainer" containerID="fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac"
Sep 18 20:06:49.873198 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.873178    2583 scope.go:117] "RemoveContainer" containerID="fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac"
Sep 18 20:06:49.873477 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:06:49.873458    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac\": container with ID starting with fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac not found: ID does not exist" containerID="fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac"
Sep 18 20:06:49.873528 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.873486    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac"} err="failed to get container status \"fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac\": rpc error: code = NotFound desc = could not find container \"fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac\": container with ID starting with fa162a5926ca2a11ced1f5f8c93734d64ed6bc584a9cde325fa8afdc49a9e8ac not found: ID does not exist"
Sep 18 20:06:49.883654 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.883633    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"]
Sep 18 20:06:49.887391 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.887338    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75lsp2p"]
Sep 18 20:06:49.965168 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:06:49.965147    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" path="/var/lib/kubelet/pods/e6671223-e56e-41c8-bba0-bc0711a1cfb7/volumes"
Sep 18 20:07:05.454008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.453975    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"]
Sep 18 20:07:05.454484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.454282    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main"
Sep 18 20:07:05.454484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.454296    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main"
Sep 18 20:07:05.454484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.454315    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635e1a68-7fa5-45f4-8610-697ad84b5e48" containerName="main"
Sep 18 20:07:05.454484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.454321    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="635e1a68-7fa5-45f4-8610-697ad84b5e48" containerName="main"
Sep 18 20:07:05.454484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.454391    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="635e1a68-7fa5-45f4-8610-697ad84b5e48" containerName="main"
Sep 18 20:07:05.454484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.454401    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6671223-e56e-41c8-bba0-bc0711a1cfb7" containerName="main"
Sep 18 20:07:05.456903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.456885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.460594 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.460559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-brqps\""
Sep 18 20:07:05.460722 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.460591    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 20:07:05.460722 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.460595    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 20:07:05.460722 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.460564    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 20:07:05.467894 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.467867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"]
Sep 18 20:07:05.548840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.548811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.548977 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.548861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.548977 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.548912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.548977 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.548933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.548977 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.548959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f215d0df-78e6-4806-9391-68a979001ef1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.549138 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.548991    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jwz\" (UniqueName: \"kubernetes.io/projected/f215d0df-78e6-4806-9391-68a979001ef1-kube-api-access-s7jwz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.650223 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f215d0df-78e6-4806-9391-68a979001ef1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.650375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jwz\" (UniqueName: \"kubernetes.io/projected/f215d0df-78e6-4806-9391-68a979001ef1-kube-api-access-s7jwz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.650375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.650499 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.650499 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.650499 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.650826 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.651177 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.650923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.651299 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.651051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.652782 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.652758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.653046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.653023    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f215d0df-78e6-4806-9391-68a979001ef1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.662312 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.662293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jwz\" (UniqueName: \"kubernetes.io/projected/f215d0df-78e6-4806-9391-68a979001ef1-kube-api-access-s7jwz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:05.768168 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:05.768135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:06.036405 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:06.036316    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"]
Sep 18 20:07:06.039272 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:07:06.039245    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf215d0df_78e6_4806_9391_68a979001ef1.slice/crio-9b48c6ab1c8a395f0296cbaf1ccdef8e971c3bed42dc37baa151cec7e00447e0 WatchSource:0}: Error finding container 9b48c6ab1c8a395f0296cbaf1ccdef8e971c3bed42dc37baa151cec7e00447e0: Status 404 returned error can't find the container with id 9b48c6ab1c8a395f0296cbaf1ccdef8e971c3bed42dc37baa151cec7e00447e0
Sep 18 20:07:06.932920 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:06.932878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" event={"ID":"f215d0df-78e6-4806-9391-68a979001ef1","Type":"ContainerStarted","Data":"2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d"}
Sep 18 20:07:06.933424 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:06.932935    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" event={"ID":"f215d0df-78e6-4806-9391-68a979001ef1","Type":"ContainerStarted","Data":"9b48c6ab1c8a395f0296cbaf1ccdef8e971c3bed42dc37baa151cec7e00447e0"}
Sep 18 20:07:06.950453 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:06.950407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" podStartSLOduration=1.950393196 podStartE2EDuration="1.950393196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:06.948142788 +0000 UTC m=+1721.202349407" watchObservedRunningTime="2026-09-18 20:07:06.950393196 +0000 UTC m=+1721.204599800"
Sep 18 20:07:07.768240 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:07.768204    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:15.768465 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:15.768429    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:15.780906 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:15.780875    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:15.980970 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:15.980939    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:49.794661 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.794622    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"]
Sep 18 20:07:49.797179 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.797158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.799846 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.799814    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 18 20:07:49.799846 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.799816    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 18 20:07:49.800728 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.800706    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 18 20:07:49.800853 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.800743    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-4np2k\""
Sep 18 20:07:49.807544 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.807520    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"]
Sep 18 20:07:49.886236 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.886207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.886421 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.886247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-kube-api-access-qvmkn\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.886421 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.886304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.886421 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.886329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.886421 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.886380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.886581 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.886426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987163 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987163 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987435 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987435 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987435 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987435 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-kube-api-access-qvmkn\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987673 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987750 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987720    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.987864 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.987733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.989635 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.989621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.989737 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.989721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:49.995940 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:49.995915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-kube-api-access-qvmkn\") pod \"test-div-v2-kserve-5f6fdb6b77-rc5hb\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:50.108551 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:50.108466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:50.378380 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:50.378335    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"]
Sep 18 20:07:50.380769 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:07:50.380740    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f65d6d3_b2f0_4d6a_ba22_264165df49dd.slice/crio-2e675c42139a50326860e1ab5ffdf7a64a437097555b7066df27360c82aa67b1 WatchSource:0}: Error finding container 2e675c42139a50326860e1ab5ffdf7a64a437097555b7066df27360c82aa67b1: Status 404 returned error can't find the container with id 2e675c42139a50326860e1ab5ffdf7a64a437097555b7066df27360c82aa67b1
Sep 18 20:07:51.105919 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:51.105878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb" event={"ID":"8f65d6d3-b2f0-4d6a-ba22-264165df49dd","Type":"ContainerStarted","Data":"2e675c42139a50326860e1ab5ffdf7a64a437097555b7066df27360c82aa67b1"}
Sep 18 20:07:51.927881 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:51.927851    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"]
Sep 18 20:07:51.936574 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:07:51.936542    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:07:51.936574 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:07:51.936574 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:07:51.936574 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d" cmd=["/bin/sleep","15"]
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:51.936588    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" message="" err=<
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:07:51.936602    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" podUID="f215d0df-78e6-4806-9391-68a979001ef1" containerName="main" containerID="cri-o://2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d"
Sep 18 20:07:51.936831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:51.936645    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" podUID="f215d0df-78e6-4806-9391-68a979001ef1" containerName="main" containerID="cri-o://2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d" gracePeriod=60
Sep 18 20:07:51.970640 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:51.970609    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" podUID="f215d0df-78e6-4806-9391-68a979001ef1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 18 20:07:52.114902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.114864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb" event={"ID":"8f65d6d3-b2f0-4d6a-ba22-264165df49dd","Type":"ContainerStarted","Data":"e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780"}
Sep 18 20:07:52.131583 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.131539    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb" podStartSLOduration=2.145146025 podStartE2EDuration="3.131525458s" totalImagesPullingTime="986.379433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:52.129896499 +0000 UTC m=+1766.384103116" watchObservedRunningTime="2026-09-18 20:07:52.131525458 +0000 UTC m=+1766.385732155"
Sep 18 20:07:52.271581 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.271559    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:52.409071 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-model-cache\") pod \"f215d0df-78e6-4806-9391-68a979001ef1\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") "
Sep 18 20:07:52.409252 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-dshm\") pod \"f215d0df-78e6-4806-9391-68a979001ef1\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") "
Sep 18 20:07:52.409252 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7jwz\" (UniqueName: \"kubernetes.io/projected/f215d0df-78e6-4806-9391-68a979001ef1-kube-api-access-s7jwz\") pod \"f215d0df-78e6-4806-9391-68a979001ef1\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") "
Sep 18 20:07:52.409252 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-tmp-dir\") pod \"f215d0df-78e6-4806-9391-68a979001ef1\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") "
Sep 18 20:07:52.409252 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f215d0df-78e6-4806-9391-68a979001ef1-tls-certs\") pod \"f215d0df-78e6-4806-9391-68a979001ef1\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") "
Sep 18 20:07:52.409252 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409229    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-home\") pod \"f215d0df-78e6-4806-9391-68a979001ef1\" (UID: \"f215d0df-78e6-4806-9391-68a979001ef1\") "
Sep 18 20:07:52.409544 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f215d0df-78e6-4806-9391-68a979001ef1" (UID: "f215d0df-78e6-4806-9391-68a979001ef1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.409544 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f215d0df-78e6-4806-9391-68a979001ef1" (UID: "f215d0df-78e6-4806-9391-68a979001ef1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.409544 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409499    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.409665 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.409543    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-home" (OuterVolumeSpecName: "home") pod "f215d0df-78e6-4806-9391-68a979001ef1" (UID: "f215d0df-78e6-4806-9391-68a979001ef1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.411563 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.411539    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-dshm" (OuterVolumeSpecName: "dshm") pod "f215d0df-78e6-4806-9391-68a979001ef1" (UID: "f215d0df-78e6-4806-9391-68a979001ef1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.411679 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.411621    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f215d0df-78e6-4806-9391-68a979001ef1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f215d0df-78e6-4806-9391-68a979001ef1" (UID: "f215d0df-78e6-4806-9391-68a979001ef1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:52.411679 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.411660    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f215d0df-78e6-4806-9391-68a979001ef1-kube-api-access-s7jwz" (OuterVolumeSpecName: "kube-api-access-s7jwz") pod "f215d0df-78e6-4806-9391-68a979001ef1" (UID: "f215d0df-78e6-4806-9391-68a979001ef1"). InnerVolumeSpecName "kube-api-access-s7jwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:52.510063 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.510043    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.510179 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.510066    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jwz\" (UniqueName: \"kubernetes.io/projected/f215d0df-78e6-4806-9391-68a979001ef1-kube-api-access-s7jwz\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.510179 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.510076    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.510179 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.510086    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f215d0df-78e6-4806-9391-68a979001ef1-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.510179 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:52.510093    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f215d0df-78e6-4806-9391-68a979001ef1-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:07:53.109003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.108969    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:07:53.119858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.119824    2583 generic.go:356] "Generic (PLEG): container finished" podID="f215d0df-78e6-4806-9391-68a979001ef1" containerID="2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d" exitCode=0
Sep 18 20:07:53.120279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.119887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" event={"ID":"f215d0df-78e6-4806-9391-68a979001ef1","Type":"ContainerDied","Data":"2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d"}
Sep 18 20:07:53.120279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.119905    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"
Sep 18 20:07:53.120279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.119944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh" event={"ID":"f215d0df-78e6-4806-9391-68a979001ef1","Type":"ContainerDied","Data":"9b48c6ab1c8a395f0296cbaf1ccdef8e971c3bed42dc37baa151cec7e00447e0"}
Sep 18 20:07:53.120279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.119970    2583 scope.go:117] "RemoveContainer" containerID="2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d"
Sep 18 20:07:53.129005 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.128986    2583 scope.go:117] "RemoveContainer" containerID="2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d"
Sep 18 20:07:53.129258 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:07:53.129241    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d\": container with ID starting with 2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d not found: ID does not exist" containerID="2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d"
Sep 18 20:07:53.129318 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.129266    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d"} err="failed to get container status \"2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d\": rpc error: code = NotFound desc = could not find container \"2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d\": container with ID starting with 2d59cbf7814d67e4cd295b4461137a321154689df4a4258f453338a563d68f5d not found: ID does not exist"
Sep 18 20:07:53.141809 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.141766    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"]
Sep 18 20:07:53.142969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.142948    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ds7kh"]
Sep 18 20:07:53.965799 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:07:53.965767    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f215d0df-78e6-4806-9391-68a979001ef1" path="/var/lib/kubelet/pods/f215d0df-78e6-4806-9391-68a979001ef1/volumes"
Sep 18 20:08:00.109227 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:00.109187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:08:00.121743 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:00.121721    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:08:00.161970 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:00.161944    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:08:27.972967 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:27.972930    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:08:27.977823 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:27.977803    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:08:39.404620 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:39.404587    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"]
Sep 18 20:08:54.424542 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.424503    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb" podUID="8f65d6d3-b2f0-4d6a-ba22-264165df49dd" containerName="main" containerID="cri-o://e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780" gracePeriod=45
Sep 18 20:08:54.654535 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.654508    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:08:54.773369 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-kube-api-access-qvmkn\") pod \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") "
Sep 18 20:08:54.773495 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773387    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-model-cache\") pod \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") "
Sep 18 20:08:54.773495 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773413    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tls-certs\") pod \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") "
Sep 18 20:08:54.773495 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773463    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tmp-dir\") pod \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") "
Sep 18 20:08:54.773664 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-home\") pod \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") "
Sep 18 20:08:54.773664 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-dshm\") pod \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\" (UID: \"8f65d6d3-b2f0-4d6a-ba22-264165df49dd\") "
Sep 18 20:08:54.773664 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773624    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f65d6d3-b2f0-4d6a-ba22-264165df49dd" (UID: "8f65d6d3-b2f0-4d6a-ba22-264165df49dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.773822 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773729    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f65d6d3-b2f0-4d6a-ba22-264165df49dd" (UID: "8f65d6d3-b2f0-4d6a-ba22-264165df49dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.773822 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773743    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-home" (OuterVolumeSpecName: "home") pod "8f65d6d3-b2f0-4d6a-ba22-264165df49dd" (UID: "8f65d6d3-b2f0-4d6a-ba22-264165df49dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.773822 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.773786    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.775721 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.775703    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f65d6d3-b2f0-4d6a-ba22-264165df49dd" (UID: "8f65d6d3-b2f0-4d6a-ba22-264165df49dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:08:54.775801 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.775742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-kube-api-access-qvmkn" (OuterVolumeSpecName: "kube-api-access-qvmkn") pod "8f65d6d3-b2f0-4d6a-ba22-264165df49dd" (UID: "8f65d6d3-b2f0-4d6a-ba22-264165df49dd"). InnerVolumeSpecName "kube-api-access-qvmkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:54.775801 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.775791    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-dshm" (OuterVolumeSpecName: "dshm") pod "8f65d6d3-b2f0-4d6a-ba22-264165df49dd" (UID: "8f65d6d3-b2f0-4d6a-ba22-264165df49dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.874486 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.874463    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmkn\" (UniqueName: \"kubernetes.io/projected/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-kube-api-access-qvmkn\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.874486 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.874486    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.874603 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.874514    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.874603 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.874522    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.874603 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:54.874529    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f65d6d3-b2f0-4d6a-ba22-264165df49dd-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:08:55.370838 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.370799    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f65d6d3-b2f0-4d6a-ba22-264165df49dd" containerID="e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780" exitCode=0
Sep 18 20:08:55.371025 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.370850    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"
Sep 18 20:08:55.371025 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.370855    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb" event={"ID":"8f65d6d3-b2f0-4d6a-ba22-264165df49dd","Type":"ContainerDied","Data":"e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780"}
Sep 18 20:08:55.371025 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.370916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb" event={"ID":"8f65d6d3-b2f0-4d6a-ba22-264165df49dd","Type":"ContainerDied","Data":"2e675c42139a50326860e1ab5ffdf7a64a437097555b7066df27360c82aa67b1"}
Sep 18 20:08:55.371025 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.370946    2583 scope.go:117] "RemoveContainer" containerID="e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780"
Sep 18 20:08:55.379216 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.379193    2583 scope.go:117] "RemoveContainer" containerID="e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780"
Sep 18 20:08:55.379510 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:08:55.379492    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780\": container with ID starting with e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780 not found: ID does not exist" containerID="e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780"
Sep 18 20:08:55.379570 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.379520    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780"} err="failed to get container status \"e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780\": rpc error: code = NotFound desc = could not find container \"e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780\": container with ID starting with e882542f33c301400ec16c0dbe08a7f1c4edddb6b491aeeb83837707e0684780 not found: ID does not exist"
Sep 18 20:08:55.390407 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.390384    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"]
Sep 18 20:08:55.394546 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.394523    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-rc5hb"]
Sep 18 20:08:55.965971 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:08:55.965942    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f65d6d3-b2f0-4d6a-ba22-264165df49dd" path="/var/lib/kubelet/pods/8f65d6d3-b2f0-4d6a-ba22-264165df49dd/volumes"
Sep 18 20:09:21.176786 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.176752    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"]
Sep 18 20:09:21.177322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.177164    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f215d0df-78e6-4806-9391-68a979001ef1" containerName="main"
Sep 18 20:09:21.177322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.177180    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f215d0df-78e6-4806-9391-68a979001ef1" containerName="main"
Sep 18 20:09:21.177322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.177203    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f65d6d3-b2f0-4d6a-ba22-264165df49dd" containerName="main"
Sep 18 20:09:21.177322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.177211    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f65d6d3-b2f0-4d6a-ba22-264165df49dd" containerName="main"
Sep 18 20:09:21.177322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.177283    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f65d6d3-b2f0-4d6a-ba22-264165df49dd" containerName="main"
Sep 18 20:09:21.177322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.177296    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f215d0df-78e6-4806-9391-68a979001ef1" containerName="main"
Sep 18 20:09:21.180424 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.180404    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.183423 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.183402    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:09:21.184331 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.184308    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 20:09:21.184454 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.184441    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 20:09:21.184602 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.184584    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 20:09:21.184703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.184630    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 20:09:21.184703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.184691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-ghbfp\""
Sep 18 20:09:21.189623 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.189605    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"]
Sep 18 20:09:21.273190 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-dshm\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.273190 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7xb\" (UniqueName: \"kubernetes.io/projected/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kube-api-access-nt7xb\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.273425 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.273425 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-home\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.273425 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.273425 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.273587 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.273587 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.273489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7347189b-ab26-4ad7-bec8-fc5623f1af8e-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.374602 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-dshm\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.374739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7xb\" (UniqueName: \"kubernetes.io/projected/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kube-api-access-nt7xb\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.374739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.374739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-home\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.374739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.374739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.375019 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.375019 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.374808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7347189b-ab26-4ad7-bec8-fc5623f1af8e-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.375132 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.375092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-home\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.375187 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.375125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.375187 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.375152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.375187 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.375181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.375547 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.375525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7347189b-ab26-4ad7-bec8-fc5623f1af8e-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.377542 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.377521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-dshm\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.377701 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.377684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.382481 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.382464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7xb\" (UniqueName: \"kubernetes.io/projected/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kube-api-access-nt7xb\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.491218 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.491133    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:21.761210 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:21.761174    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"]
Sep 18 20:09:21.764605 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:09:21.764577    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7347189b_ab26_4ad7_bec8_fc5623f1af8e.slice/crio-8fcfecfb64acbacfa63af2e7bf25728e0a6e1fba44e5cf31a94bb7bb012e2d6a WatchSource:0}: Error finding container 8fcfecfb64acbacfa63af2e7bf25728e0a6e1fba44e5cf31a94bb7bb012e2d6a: Status 404 returned error can't find the container with id 8fcfecfb64acbacfa63af2e7bf25728e0a6e1fba44e5cf31a94bb7bb012e2d6a
Sep 18 20:09:22.485803 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:22.485768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" event={"ID":"7347189b-ab26-4ad7-bec8-fc5623f1af8e","Type":"ContainerStarted","Data":"727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274"}
Sep 18 20:09:22.486176 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:22.485829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" event={"ID":"7347189b-ab26-4ad7-bec8-fc5623f1af8e","Type":"ContainerStarted","Data":"8fcfecfb64acbacfa63af2e7bf25728e0a6e1fba44e5cf31a94bb7bb012e2d6a"}
Sep 18 20:09:24.496458 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:24.496422    2583 generic.go:356] "Generic (PLEG): container finished" podID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerID="727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274" exitCode=0
Sep 18 20:09:24.496820 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:24.496494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" event={"ID":"7347189b-ab26-4ad7-bec8-fc5623f1af8e","Type":"ContainerDied","Data":"727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274"}
Sep 18 20:09:25.502420 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:25.502384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" event={"ID":"7347189b-ab26-4ad7-bec8-fc5623f1af8e","Type":"ContainerStarted","Data":"11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97"}
Sep 18 20:09:25.521965 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:25.521886    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" podStartSLOduration=2.3502159049999998 podStartE2EDuration="4.521868168s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.171652263s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:25.521140185 +0000 UTC m=+1859.775346792" watchObservedRunningTime="2026-09-18 20:09:25.521868168 +0000 UTC m=+1859.776074773"
Sep 18 20:09:26.492243 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:26.492194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:31.492256 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:31.492217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:31.504269 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:31.504240    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:09:31.544485 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:09:31.544461    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:10:21.799636 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.799604    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"]
Sep 18 20:10:21.802779 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.802763    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:21.806228 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.806204    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 18 20:10:21.806382 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.806240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-btwqd\""
Sep 18 20:10:21.806382 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.806270    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 18 20:10:21.806382 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.806367    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 18 20:10:21.811619 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.811598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"]
Sep 18 20:10:21.934627 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.934591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvckp\" (UniqueName: \"kubernetes.io/projected/33cc0306-a460-413b-b0dd-3fe700860c89-kube-api-access-zvckp\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:21.934627 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.934629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:21.934810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.934649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33cc0306-a460-413b-b0dd-3fe700860c89-tls-certs\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:21.934810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.934683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-home\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:21.934810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.934728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-model-cache\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:21.934810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:21.934766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-dshm\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.035810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.035779    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-dshm\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.035989 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.035822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvckp\" (UniqueName: \"kubernetes.io/projected/33cc0306-a460-413b-b0dd-3fe700860c89-kube-api-access-zvckp\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.035989 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.035840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.035989 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.035854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33cc0306-a460-413b-b0dd-3fe700860c89-tls-certs\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.035989 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.035868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-home\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.035989 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.035895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-model-cache\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.036270 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.036252    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-home\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.036326 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.036291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.036326 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.036302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-model-cache\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.038247 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.038219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-dshm\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.038538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.038518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33cc0306-a460-413b-b0dd-3fe700860c89-tls-certs\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.043776 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.043751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvckp\" (UniqueName: \"kubernetes.io/projected/33cc0306-a460-413b-b0dd-3fe700860c89-kube-api-access-zvckp\") pod \"stop-v1-kserve-5657b88c95-mccxb\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.114039 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.113982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:22.405292 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.405266    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"]
Sep 18 20:10:22.407678 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:10:22.407651    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cc0306_a460_413b_b0dd_3fe700860c89.slice/crio-a4b73aa6a72fc17f51a417dee97a8a884fa5b2a7248ede1ea932576d3987c828 WatchSource:0}: Error finding container a4b73aa6a72fc17f51a417dee97a8a884fa5b2a7248ede1ea932576d3987c828: Status 404 returned error can't find the container with id a4b73aa6a72fc17f51a417dee97a8a884fa5b2a7248ede1ea932576d3987c828
Sep 18 20:10:22.741510 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.741413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb" event={"ID":"33cc0306-a460-413b-b0dd-3fe700860c89","Type":"ContainerStarted","Data":"ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9"}
Sep 18 20:10:22.741510 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.741480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb" event={"ID":"33cc0306-a460-413b-b0dd-3fe700860c89","Type":"ContainerStarted","Data":"a4b73aa6a72fc17f51a417dee97a8a884fa5b2a7248ede1ea932576d3987c828"}
Sep 18 20:10:22.759246 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:22.759198    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb" podStartSLOduration=1.759180287 podStartE2EDuration="1.759180287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:22.75773331 +0000 UTC m=+1917.011939922" watchObservedRunningTime="2026-09-18 20:10:22.759180287 +0000 UTC m=+1917.013386892"
Sep 18 20:10:23.114684 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:23.114655    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:32.114324 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:32.114291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:32.126496 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:32.126476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:32.792189 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:32.792164    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:10:59.828476 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:59.828443    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"]
Sep 18 20:10:59.837830 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:10:59.837793    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:10:59.837830 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:10:59.837830 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:10:59.837830 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97" cmd=["/bin/sleep","15"]
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:59.837853    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" message="" err=<
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:10:59.837870    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="main" containerID="cri-o://11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97"
Sep 18 20:10:59.838085 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:10:59.837922    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="main" containerID="cri-o://11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97" gracePeriod=60
Sep 18 20:11:00.581953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.581931    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:11:00.622049 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7347189b-ab26-4ad7-bec8-fc5623f1af8e-cabundle-cert\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622194 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622053    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-home\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622194 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622086    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kserve-provision-location\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622194 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-dshm\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622194 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622170    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tls-certs\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622200    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt7xb\" (UniqueName: \"kubernetes.io/projected/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kube-api-access-nt7xb\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622233    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tmp-dir\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622265    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-model-cache\") pod \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\" (UID: \"7347189b-ab26-4ad7-bec8-fc5623f1af8e\") "
Sep 18 20:11:00.622375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622317    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-home" (OuterVolumeSpecName: "home") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.622522 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622437    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7347189b-ab26-4ad7-bec8-fc5623f1af8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:11:00.622557 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622537    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.622661 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622644    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.622705 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622668    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7347189b-ab26-4ad7-bec8-fc5623f1af8e-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.622705 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622681    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.622946 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.622919    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.624646 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.624620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:11:00.624749 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.624694    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-dshm" (OuterVolumeSpecName: "dshm") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.625010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.624979    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kube-api-access-nt7xb" (OuterVolumeSpecName: "kube-api-access-nt7xb") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "kube-api-access-nt7xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:00.642628 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.642604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7347189b-ab26-4ad7-bec8-fc5623f1af8e" (UID: "7347189b-ab26-4ad7-bec8-fc5623f1af8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.723525 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.723469    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt7xb\" (UniqueName: \"kubernetes.io/projected/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kube-api-access-nt7xb\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.723525 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.723508    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.723525 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.723519    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.723525 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.723527    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7347189b-ab26-4ad7-bec8-fc5623f1af8e-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.723712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.723535    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7347189b-ab26-4ad7-bec8-fc5623f1af8e-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.897163 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.897128    2583 generic.go:356] "Generic (PLEG): container finished" podID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerID="11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97" exitCode=0
Sep 18 20:11:00.897553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.897180    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"
Sep 18 20:11:00.897553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.897216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" event={"ID":"7347189b-ab26-4ad7-bec8-fc5623f1af8e","Type":"ContainerDied","Data":"11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97"}
Sep 18 20:11:00.897553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.897276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" event={"ID":"7347189b-ab26-4ad7-bec8-fc5623f1af8e","Type":"ContainerDied","Data":"8fcfecfb64acbacfa63af2e7bf25728e0a6e1fba44e5cf31a94bb7bb012e2d6a"}
Sep 18 20:11:00.897553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.897307    2583 scope.go:117] "RemoveContainer" containerID="11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97"
Sep 18 20:11:00.906774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.906760    2583 scope.go:117] "RemoveContainer" containerID="727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274"
Sep 18 20:11:00.913939 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.913923    2583 scope.go:117] "RemoveContainer" containerID="11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97"
Sep 18 20:11:00.914173 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:11:00.914156    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97\": container with ID starting with 11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97 not found: ID does not exist" containerID="11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97"
Sep 18 20:11:00.914221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.914180    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97"} err="failed to get container status \"11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97\": rpc error: code = NotFound desc = could not find container \"11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97\": container with ID starting with 11b25d28aba96f019f13cc52b7705141d1179f9bea9be5fc1c2ddbfe50b78e97 not found: ID does not exist"
Sep 18 20:11:00.914221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.914196    2583 scope.go:117] "RemoveContainer" containerID="727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274"
Sep 18 20:11:00.914405 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:11:00.914387    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274\": container with ID starting with 727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274 not found: ID does not exist" containerID="727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274"
Sep 18 20:11:00.914450 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.914409    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274"} err="failed to get container status \"727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274\": rpc error: code = NotFound desc = could not find container \"727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274\": container with ID starting with 727b1bde88245dd586ff3ff289e9610eb1773726439d8c94d520e4ec40747274 not found: ID does not exist"
Sep 18 20:11:00.919507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.919449    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"]
Sep 18 20:11:00.920777 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:00.920755    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq"]
Sep 18 20:11:01.534641 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:01.534602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-7q7mq" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:11:01.967106 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:01.967028    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" path="/var/lib/kubelet/pods/7347189b-ab26-4ad7-bec8-fc5623f1af8e/volumes"
Sep 18 20:11:40.086074 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:40.086044    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"]
Sep 18 20:11:55.105288 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.105245    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb" podUID="33cc0306-a460-413b-b0dd-3fe700860c89" containerName="main" containerID="cri-o://ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9" gracePeriod=45
Sep 18 20:11:55.347221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.347196    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:11:55.413204 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-home\") pod \"33cc0306-a460-413b-b0dd-3fe700860c89\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") "
Sep 18 20:11:55.413204 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-model-cache\") pod \"33cc0306-a460-413b-b0dd-3fe700860c89\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") "
Sep 18 20:11:55.413204 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413189    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-tmp-dir\") pod \"33cc0306-a460-413b-b0dd-3fe700860c89\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") "
Sep 18 20:11:55.413491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvckp\" (UniqueName: \"kubernetes.io/projected/33cc0306-a460-413b-b0dd-3fe700860c89-kube-api-access-zvckp\") pod \"33cc0306-a460-413b-b0dd-3fe700860c89\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") "
Sep 18 20:11:55.413491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413232    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-dshm\") pod \"33cc0306-a460-413b-b0dd-3fe700860c89\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") "
Sep 18 20:11:55.413491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33cc0306-a460-413b-b0dd-3fe700860c89-tls-certs\") pod \"33cc0306-a460-413b-b0dd-3fe700860c89\" (UID: \"33cc0306-a460-413b-b0dd-3fe700860c89\") "
Sep 18 20:11:55.413491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413433    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-home" (OuterVolumeSpecName: "home") pod "33cc0306-a460-413b-b0dd-3fe700860c89" (UID: "33cc0306-a460-413b-b0dd-3fe700860c89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:55.413491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33cc0306-a460-413b-b0dd-3fe700860c89" (UID: "33cc0306-a460-413b-b0dd-3fe700860c89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:55.413740 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413506    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-model-cache" (OuterVolumeSpecName: "model-cache") pod "33cc0306-a460-413b-b0dd-3fe700860c89" (UID: "33cc0306-a460-413b-b0dd-3fe700860c89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:55.413740 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413697    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:55.413740 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413716    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:55.413740 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.413729    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:55.415627 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.415605    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-dshm" (OuterVolumeSpecName: "dshm") pod "33cc0306-a460-413b-b0dd-3fe700860c89" (UID: "33cc0306-a460-413b-b0dd-3fe700860c89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:55.415708 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.415664    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33cc0306-a460-413b-b0dd-3fe700860c89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33cc0306-a460-413b-b0dd-3fe700860c89" (UID: "33cc0306-a460-413b-b0dd-3fe700860c89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:11:55.415759 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.415744    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cc0306-a460-413b-b0dd-3fe700860c89-kube-api-access-zvckp" (OuterVolumeSpecName: "kube-api-access-zvckp") pod "33cc0306-a460-413b-b0dd-3fe700860c89" (UID: "33cc0306-a460-413b-b0dd-3fe700860c89"). InnerVolumeSpecName "kube-api-access-zvckp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:55.514836 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.514800    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvckp\" (UniqueName: \"kubernetes.io/projected/33cc0306-a460-413b-b0dd-3fe700860c89-kube-api-access-zvckp\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:55.514836 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.514827    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33cc0306-a460-413b-b0dd-3fe700860c89-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:55.514836 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:55.514836    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33cc0306-a460-413b-b0dd-3fe700860c89-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:11:56.109882 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.109841    2583 generic.go:356] "Generic (PLEG): container finished" podID="33cc0306-a460-413b-b0dd-3fe700860c89" containerID="ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9" exitCode=0
Sep 18 20:11:56.109882 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.109884    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"
Sep 18 20:11:56.110448 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.109923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb" event={"ID":"33cc0306-a460-413b-b0dd-3fe700860c89","Type":"ContainerDied","Data":"ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9"}
Sep 18 20:11:56.110448 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.110010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb" event={"ID":"33cc0306-a460-413b-b0dd-3fe700860c89","Type":"ContainerDied","Data":"a4b73aa6a72fc17f51a417dee97a8a884fa5b2a7248ede1ea932576d3987c828"}
Sep 18 20:11:56.110448 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.110036    2583 scope.go:117] "RemoveContainer" containerID="ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9"
Sep 18 20:11:56.118015 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.117998    2583 scope.go:117] "RemoveContainer" containerID="ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9"
Sep 18 20:11:56.118272 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:11:56.118254    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9\": container with ID starting with ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9 not found: ID does not exist" containerID="ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9"
Sep 18 20:11:56.118319 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.118280    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9"} err="failed to get container status \"ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9\": rpc error: code = NotFound desc = could not find container \"ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9\": container with ID starting with ea9564206699430a797d6c3bf76c1a96cad6dded877a13182c05a3b9cc4d90d9 not found: ID does not exist"
Sep 18 20:11:56.125163 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.125142    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"]
Sep 18 20:11:56.131797 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:56.131778    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mccxb"]
Sep 18 20:11:57.966678 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:11:57.966644    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33cc0306-a460-413b-b0dd-3fe700860c89" path="/var/lib/kubelet/pods/33cc0306-a460-413b-b0dd-3fe700860c89/volumes"
Sep 18 20:12:00.510319 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510284    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"]
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510580    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="main"
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510593    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="main"
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510607    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="storage-initializer"
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510613    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="storage-initializer"
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510633    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cc0306-a460-413b-b0dd-3fe700860c89" containerName="main"
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510640    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cc0306-a460-413b-b0dd-3fe700860c89" containerName="main"
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510686    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7347189b-ab26-4ad7-bec8-fc5623f1af8e" containerName="main"
Sep 18 20:12:00.510706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.510693    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="33cc0306-a460-413b-b0dd-3fe700860c89" containerName="main"
Sep 18 20:12:00.514869 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.514850    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.518423 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.518396    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 18 20:12:00.519375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.519340    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-lcmpn\""
Sep 18 20:12:00.519553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.519537    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 18 20:12:00.519660 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.519644    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 18 20:12:00.527768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.527741    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"]
Sep 18 20:12:00.656275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.656244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-tmp-dir\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.656275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.656278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-home\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.656478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.656344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6992f2c1-a607-427a-88ab-2bd1be2220d7-tls-certs\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.656478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.656388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-dshm\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.656478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.656404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-model-cache\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.656478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.656424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz7v\" (UniqueName: \"kubernetes.io/projected/6992f2c1-a607-427a-88ab-2bd1be2220d7-kube-api-access-dkz7v\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.757583 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.757553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-tmp-dir\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.757702 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.757588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-home\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.757744 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.757713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6992f2c1-a607-427a-88ab-2bd1be2220d7-tls-certs\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.757779 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.757747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-dshm\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.757779 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.757764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-model-cache\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.757859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.757792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz7v\" (UniqueName: \"kubernetes.io/projected/6992f2c1-a607-427a-88ab-2bd1be2220d7-kube-api-access-dkz7v\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.758003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.757983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-tmp-dir\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.758052 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.758032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-home\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.758095 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.758056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-model-cache\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.760119 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.760094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-dshm\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.760453 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.760413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6992f2c1-a607-427a-88ab-2bd1be2220d7-tls-certs\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.769722 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.769695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz7v\" (UniqueName: \"kubernetes.io/projected/6992f2c1-a607-427a-88ab-2bd1be2220d7-kube-api-access-dkz7v\") pod \"decom-v2-kserve-77f867d787-b8wvm\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:00.825600 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:00.825578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:01.099797 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:01.099671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"]
Sep 18 20:12:01.101881 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:12:01.101857    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6992f2c1_a607_427a_88ab_2bd1be2220d7.slice/crio-398c81f7d20d774c432a95c6ddd2e2dfde9cd0227184f1d224725c21a802c1ff WatchSource:0}: Error finding container 398c81f7d20d774c432a95c6ddd2e2dfde9cd0227184f1d224725c21a802c1ff: Status 404 returned error can't find the container with id 398c81f7d20d774c432a95c6ddd2e2dfde9cd0227184f1d224725c21a802c1ff
Sep 18 20:12:01.134046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:01.134022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm" event={"ID":"6992f2c1-a607-427a-88ab-2bd1be2220d7","Type":"ContainerStarted","Data":"398c81f7d20d774c432a95c6ddd2e2dfde9cd0227184f1d224725c21a802c1ff"}
Sep 18 20:12:02.139743 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:02.139704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm" event={"ID":"6992f2c1-a607-427a-88ab-2bd1be2220d7","Type":"ContainerStarted","Data":"dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55"}
Sep 18 20:12:02.162026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:02.161982    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm" podStartSLOduration=2.161967454 podStartE2EDuration="2.161967454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:02.161172253 +0000 UTC m=+2016.415378858" watchObservedRunningTime="2026-09-18 20:12:02.161967454 +0000 UTC m=+2016.416174058"
Sep 18 20:12:02.826472 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:02.826441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:10.826531 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:10.826492    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:10.838989 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:10.838958    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:11.187555 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:11.187473    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:12:13.494489 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.494455    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"]
Sep 18 20:12:13.497556 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.497535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.500409 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.500388    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 20:12:13.500627 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.500607    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:12:13.501539 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.501517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-7q8zm\""
Sep 18 20:12:13.501752 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.501733    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 20:12:13.501854 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.501837    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 20:12:13.501927 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.501855    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 20:12:13.510401 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.510330    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"]
Sep 18 20:12:13.659521 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-dshm\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.659651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659533    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dn6w\" (UniqueName: \"kubernetes.io/projected/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kube-api-access-5dn6w\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.659651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.659651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-home\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.659651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1911f5f3-2af6-4d1a-9666-446b3be56e9a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.659651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.659825 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.659825 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.659743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761104 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761231 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-home\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761231 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1911f5f3-2af6-4d1a-9666-446b3be56e9a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761231 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761231 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761231 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-dshm\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dn6w\" (UniqueName: \"kubernetes.io/projected/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kube-api-access-5dn6w\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-home\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761735 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.761735 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.761698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.762116 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.762090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1911f5f3-2af6-4d1a-9666-446b3be56e9a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.763683 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.763662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-dshm\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.763930 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.763911    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.774651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.774627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dn6w\" (UniqueName: \"kubernetes.io/projected/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kube-api-access-5dn6w\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:13.809953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:13.809928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:14.090564 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:14.090530    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"]
Sep 18 20:12:14.092946 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:12:14.092915    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1911f5f3_2af6_4d1a_9666_446b3be56e9a.slice/crio-89d73de5c37d9f93ae637107b6b0b69b0c686b27017f1bcf8d7ae868ba9c5264 WatchSource:0}: Error finding container 89d73de5c37d9f93ae637107b6b0b69b0c686b27017f1bcf8d7ae868ba9c5264: Status 404 returned error can't find the container with id 89d73de5c37d9f93ae637107b6b0b69b0c686b27017f1bcf8d7ae868ba9c5264
Sep 18 20:12:14.189387 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:14.189334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" event={"ID":"1911f5f3-2af6-4d1a-9666-446b3be56e9a","Type":"ContainerStarted","Data":"36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43"}
Sep 18 20:12:14.189518 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:14.189416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" event={"ID":"1911f5f3-2af6-4d1a-9666-446b3be56e9a","Type":"ContainerStarted","Data":"89d73de5c37d9f93ae637107b6b0b69b0c686b27017f1bcf8d7ae868ba9c5264"}
Sep 18 20:12:17.202239 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:17.202201    2583 generic.go:356] "Generic (PLEG): container finished" podID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerID="36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43" exitCode=0
Sep 18 20:12:17.202656 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:17.202245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" event={"ID":"1911f5f3-2af6-4d1a-9666-446b3be56e9a","Type":"ContainerDied","Data":"36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43"}
Sep 18 20:12:18.208167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:18.208135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" event={"ID":"1911f5f3-2af6-4d1a-9666-446b3be56e9a","Type":"ContainerStarted","Data":"e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9"}
Sep 18 20:12:18.229887 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:18.229838    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" podStartSLOduration=3.13966881 podStartE2EDuration="5.229822197s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.090153387s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:18.227968267 +0000 UTC m=+2032.482174882" watchObservedRunningTime="2026-09-18 20:12:18.229822197 +0000 UTC m=+2032.484028802"
Sep 18 20:12:18.810639 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:18.810606    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:23.811001 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:23.810960    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:23.823632 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:23.823608    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:24.253851 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:24.253772    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:56.445539 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.445325    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"]
Sep 18 20:12:56.454969 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:12:56.454928    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:12:56.454969 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:12:56.454969 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:12:56.454969 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9" cmd=["/bin/sleep","15"]
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.454990    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" message="" err=<
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:12:56.455034    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerName="main" containerID="cri-o://e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9"
Sep 18 20:12:56.455258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.455079    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerName="main" containerID="cri-o://e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9" gracePeriod=60
Sep 18 20:12:56.793321 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.793292    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:56.888708 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-dshm\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.888858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1911f5f3-2af6-4d1a-9666-446b3be56e9a-cabundle-cert\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.888858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888758    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tmp-dir\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.888858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-model-cache\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.888858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-home\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.889052 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tls-certs\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.889052 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888885    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kserve-provision-location\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.889052 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.888921    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dn6w\" (UniqueName: \"kubernetes.io/projected/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kube-api-access-5dn6w\") pod \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\" (UID: \"1911f5f3-2af6-4d1a-9666-446b3be56e9a\") "
Sep 18 20:12:56.889206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.889066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.889206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.889079    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.889206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.889154    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-home" (OuterVolumeSpecName: "home") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.889206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.889165    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1911f5f3-2af6-4d1a-9666-446b3be56e9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:56.889420 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.889200    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.889420 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.889248    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.891380 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.891332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:56.891380 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.891339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kube-api-access-5dn6w" (OuterVolumeSpecName: "kube-api-access-5dn6w") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "kube-api-access-5dn6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:56.891544 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.891419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-dshm" (OuterVolumeSpecName: "dshm") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.909208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.909181    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1911f5f3-2af6-4d1a-9666-446b3be56e9a" (UID: "1911f5f3-2af6-4d1a-9666-446b3be56e9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.989964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.989905    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.989964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.989925    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1911f5f3-2af6-4d1a-9666-446b3be56e9a-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.989964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.989934    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.989964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.989942    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1911f5f3-2af6-4d1a-9666-446b3be56e9a-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.989964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.989951    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.989964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:56.989958    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dn6w\" (UniqueName: \"kubernetes.io/projected/1911f5f3-2af6-4d1a-9666-446b3be56e9a-kube-api-access-5dn6w\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:12:57.372187 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.372148    2583 generic.go:356] "Generic (PLEG): container finished" podID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerID="e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9" exitCode=0
Sep 18 20:12:57.372410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.372210    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" event={"ID":"1911f5f3-2af6-4d1a-9666-446b3be56e9a","Type":"ContainerDied","Data":"e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9"}
Sep 18 20:12:57.372410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.372220    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"
Sep 18 20:12:57.372410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.372264    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd" event={"ID":"1911f5f3-2af6-4d1a-9666-446b3be56e9a","Type":"ContainerDied","Data":"89d73de5c37d9f93ae637107b6b0b69b0c686b27017f1bcf8d7ae868ba9c5264"}
Sep 18 20:12:57.372410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.372292    2583 scope.go:117] "RemoveContainer" containerID="e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9"
Sep 18 20:12:57.381339 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.381314    2583 scope.go:117] "RemoveContainer" containerID="36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43"
Sep 18 20:12:57.388891 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.388872    2583 scope.go:117] "RemoveContainer" containerID="e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9"
Sep 18 20:12:57.389140 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:12:57.389120    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9\": container with ID starting with e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9 not found: ID does not exist" containerID="e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9"
Sep 18 20:12:57.389186 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.389148    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9"} err="failed to get container status \"e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9\": rpc error: code = NotFound desc = could not find container \"e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9\": container with ID starting with e2644845fcb0e2c82ef0ecea32d1e6ee6a612f11bf1e9f78656c8fec3effe4c9 not found: ID does not exist"
Sep 18 20:12:57.389186 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.389162    2583 scope.go:117] "RemoveContainer" containerID="36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43"
Sep 18 20:12:57.389377 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:12:57.389344    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43\": container with ID starting with 36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43 not found: ID does not exist" containerID="36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43"
Sep 18 20:12:57.389425 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.389385    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43"} err="failed to get container status \"36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43\": rpc error: code = NotFound desc = could not find container \"36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43\": container with ID starting with 36d8aec7ead8f71679af3e9ca54d3ba7856d9ae88860355ff5f7b18fcbec1c43 not found: ID does not exist"
Sep 18 20:12:57.396869 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.396848    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"]
Sep 18 20:12:57.401224 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.401204    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-s4bhd"]
Sep 18 20:12:57.969484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:57.969442    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" path="/var/lib/kubelet/pods/1911f5f3-2af6-4d1a-9666-446b3be56e9a/volumes"
Sep 18 20:12:58.503930 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:12:58.503887    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"]
Sep 18 20:13:13.522596 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.522525    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm" podUID="6992f2c1-a607-427a-88ab-2bd1be2220d7" containerName="main" containerID="cri-o://dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55" gracePeriod=45
Sep 18 20:13:13.769335 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.769310    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:13:13.910477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910407    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-model-cache\") pod \"6992f2c1-a607-427a-88ab-2bd1be2220d7\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") "
Sep 18 20:13:13.910477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910456    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-home\") pod \"6992f2c1-a607-427a-88ab-2bd1be2220d7\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") "
Sep 18 20:13:13.910697 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkz7v\" (UniqueName: \"kubernetes.io/projected/6992f2c1-a607-427a-88ab-2bd1be2220d7-kube-api-access-dkz7v\") pod \"6992f2c1-a607-427a-88ab-2bd1be2220d7\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") "
Sep 18 20:13:13.910697 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910558    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-tmp-dir\") pod \"6992f2c1-a607-427a-88ab-2bd1be2220d7\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") "
Sep 18 20:13:13.910697 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910591    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6992f2c1-a607-427a-88ab-2bd1be2220d7-tls-certs\") pod \"6992f2c1-a607-427a-88ab-2bd1be2220d7\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") "
Sep 18 20:13:13.910697 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910617    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-dshm\") pod \"6992f2c1-a607-427a-88ab-2bd1be2220d7\" (UID: \"6992f2c1-a607-427a-88ab-2bd1be2220d7\") "
Sep 18 20:13:13.910917 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910686    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "6992f2c1-a607-427a-88ab-2bd1be2220d7" (UID: "6992f2c1-a607-427a-88ab-2bd1be2220d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:13.910917 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910749    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-home" (OuterVolumeSpecName: "home") pod "6992f2c1-a607-427a-88ab-2bd1be2220d7" (UID: "6992f2c1-a607-427a-88ab-2bd1be2220d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:13.910917 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6992f2c1-a607-427a-88ab-2bd1be2220d7" (UID: "6992f2c1-a607-427a-88ab-2bd1be2220d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:13.910917 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910883    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:13:13.911062 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.910936    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:13:13.912770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.912745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6992f2c1-a607-427a-88ab-2bd1be2220d7-kube-api-access-dkz7v" (OuterVolumeSpecName: "kube-api-access-dkz7v") pod "6992f2c1-a607-427a-88ab-2bd1be2220d7" (UID: "6992f2c1-a607-427a-88ab-2bd1be2220d7"). InnerVolumeSpecName "kube-api-access-dkz7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:13.912866 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.912771    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-dshm" (OuterVolumeSpecName: "dshm") pod "6992f2c1-a607-427a-88ab-2bd1be2220d7" (UID: "6992f2c1-a607-427a-88ab-2bd1be2220d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:13.912866 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:13.912797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6992f2c1-a607-427a-88ab-2bd1be2220d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6992f2c1-a607-427a-88ab-2bd1be2220d7" (UID: "6992f2c1-a607-427a-88ab-2bd1be2220d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:14.011967 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.011936    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkz7v\" (UniqueName: \"kubernetes.io/projected/6992f2c1-a607-427a-88ab-2bd1be2220d7-kube-api-access-dkz7v\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.011967 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.011962    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.012130 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.011971    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6992f2c1-a607-427a-88ab-2bd1be2220d7-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.012130 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.011986    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6992f2c1-a607-427a-88ab-2bd1be2220d7-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:13:14.440441 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.440405    2583 generic.go:356] "Generic (PLEG): container finished" podID="6992f2c1-a607-427a-88ab-2bd1be2220d7" containerID="dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55" exitCode=0
Sep 18 20:13:14.440586 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.440459    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"
Sep 18 20:13:14.440586 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.440485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm" event={"ID":"6992f2c1-a607-427a-88ab-2bd1be2220d7","Type":"ContainerDied","Data":"dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55"}
Sep 18 20:13:14.440586 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.440545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm" event={"ID":"6992f2c1-a607-427a-88ab-2bd1be2220d7","Type":"ContainerDied","Data":"398c81f7d20d774c432a95c6ddd2e2dfde9cd0227184f1d224725c21a802c1ff"}
Sep 18 20:13:14.440586 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.440569    2583 scope.go:117] "RemoveContainer" containerID="dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55"
Sep 18 20:13:14.450937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.450913    2583 scope.go:117] "RemoveContainer" containerID="dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55"
Sep 18 20:13:14.451472 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:13:14.451447    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55\": container with ID starting with dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55 not found: ID does not exist" containerID="dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55"
Sep 18 20:13:14.451582 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.451503    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55"} err="failed to get container status \"dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55\": rpc error: code = NotFound desc = could not find container \"dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55\": container with ID starting with dda957948bb46cf7be2ba27931d1082046a03f97b76106a8df6939c011316b55 not found: ID does not exist"
Sep 18 20:13:14.467427 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.467402    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"]
Sep 18 20:13:14.470609 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:14.470589    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-b8wvm"]
Sep 18 20:13:15.966038 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:15.966007    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6992f2c1-a607-427a-88ab-2bd1be2220d7" path="/var/lib/kubelet/pods/6992f2c1-a607-427a-88ab-2bd1be2220d7/volumes"
Sep 18 20:13:27.998419 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:27.998392    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:13:28.004026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:28.004007    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:13:30.199774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.199734    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"]
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200019    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerName="storage-initializer"
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200030    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerName="storage-initializer"
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200040    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6992f2c1-a607-427a-88ab-2bd1be2220d7" containerName="main"
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200045    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6992f2c1-a607-427a-88ab-2bd1be2220d7" containerName="main"
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200053    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerName="main"
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200058    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerName="main"
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200108    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6992f2c1-a607-427a-88ab-2bd1be2220d7" containerName="main"
Sep 18 20:13:30.200294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.200118    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1911f5f3-2af6-4d1a-9666-446b3be56e9a" containerName="main"
Sep 18 20:13:30.204093 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.204075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.209539 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.209499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 18 20:13:30.209539 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.209504    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 18 20:13:30.209727 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.209504    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-9b2rl\""
Sep 18 20:13:30.209727 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.209534    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 18 20:13:30.221762 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.221738    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"]
Sep 18 20:13:30.329615 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.329577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.329615 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.329623    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-dshm\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.329802 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.329647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7n5\" (UniqueName: \"kubernetes.io/projected/57f3d181-c53c-476c-b9a2-4ad00b104c19-kube-api-access-jl7n5\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.329802 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.329706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-model-cache\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.329802 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.329742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f3d181-c53c-476c-b9a2-4ad00b104c19-tls-certs\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.329802 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.329785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-home\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431034 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.430995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431034 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-dshm\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431241 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7n5\" (UniqueName: \"kubernetes.io/projected/57f3d181-c53c-476c-b9a2-4ad00b104c19-kube-api-access-jl7n5\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431241 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-model-cache\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431241 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f3d181-c53c-476c-b9a2-4ad00b104c19-tls-certs\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431241 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-home\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431500 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431593 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-model-cache\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.431655 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.431633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-home\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.433626 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.433609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-dshm\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.433940 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.433925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f3d181-c53c-476c-b9a2-4ad00b104c19-tls-certs\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.444550 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.444532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7n5\" (UniqueName: \"kubernetes.io/projected/57f3d181-c53c-476c-b9a2-4ad00b104c19-kube-api-access-jl7n5\") pod \"leave-v1-kserve-65684d4f5-n2hnb\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.514158 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.514139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:30.796810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:30.796737    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"]
Sep 18 20:13:30.799937 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:13:30.799910    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f3d181_c53c_476c_b9a2_4ad00b104c19.slice/crio-2be2595d10a1f2537bd9531815573fa01da870b2089fe65a534a0288898165ec WatchSource:0}: Error finding container 2be2595d10a1f2537bd9531815573fa01da870b2089fe65a534a0288898165ec: Status 404 returned error can't find the container with id 2be2595d10a1f2537bd9531815573fa01da870b2089fe65a534a0288898165ec
Sep 18 20:13:31.509723 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:31.509684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb" event={"ID":"57f3d181-c53c-476c-b9a2-4ad00b104c19","Type":"ContainerStarted","Data":"ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2"}
Sep 18 20:13:31.510097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:31.509769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb" event={"ID":"57f3d181-c53c-476c-b9a2-4ad00b104c19","Type":"ContainerStarted","Data":"2be2595d10a1f2537bd9531815573fa01da870b2089fe65a534a0288898165ec"}
Sep 18 20:13:31.514250 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:31.514230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:31.535079 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:31.535034    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb" podStartSLOduration=1.5350212170000002 podStartE2EDuration="1.535021217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:31.534163178 +0000 UTC m=+2105.788369784" watchObservedRunningTime="2026-09-18 20:13:31.535021217 +0000 UTC m=+2105.789227822"
Sep 18 20:13:40.514425 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:40.514390    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:40.526659 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:40.526633    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:13:40.556885 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:13:40.556864    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:14:19.531128 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.531094    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"]
Sep 18 20:14:19.534519 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.534501    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.544310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.544287    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 20:14:19.544455 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.544309    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 20:14:19.544455 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.544291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 20:14:19.544455 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.544291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:14:19.544797 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.544781    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 20:14:19.544844 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.544825    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-4jbhf\""
Sep 18 20:14:19.565765 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.565743    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"]
Sep 18 20:14:19.587688 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhb7\" (UniqueName: \"kubernetes.io/projected/a107d23f-45bb-4867-aa13-99032c4c77db-kube-api-access-hmhb7\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.587814 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a107d23f-45bb-4867-aa13-99032c4c77db-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.587814 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-dshm\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.587814 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.587963 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-home\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.587963 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-model-cache\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.587963 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-tmp-dir\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.587963 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.587895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a107d23f-45bb-4867-aa13-99032c4c77db-tls-certs\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689096 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-tmp-dir\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689096 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a107d23f-45bb-4867-aa13-99032c4c77db-tls-certs\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689328 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhb7\" (UniqueName: \"kubernetes.io/projected/a107d23f-45bb-4867-aa13-99032c4c77db-kube-api-access-hmhb7\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689328 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a107d23f-45bb-4867-aa13-99032c4c77db-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689328 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-dshm\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689328 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689588 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-home\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689588 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-tmp-dir\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689723 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-home\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689830 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-model-cache\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.689902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.689878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-model-cache\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.690152 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.690132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a107d23f-45bb-4867-aa13-99032c4c77db-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.691788 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.691768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-dshm\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.692018 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.691999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a107d23f-45bb-4867-aa13-99032c4c77db-tls-certs\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.703664 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.703643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhb7\" (UniqueName: \"kubernetes.io/projected/a107d23f-45bb-4867-aa13-99032c4c77db-kube-api-access-hmhb7\") pod \"tokenizer-template-args-test-kserve-649b57846b-xc968\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:19.844239 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:19.844161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:20.190529 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:20.190501    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"]
Sep 18 20:14:20.191915 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:14:20.191890    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda107d23f_45bb_4867_aa13_99032c4c77db.slice/crio-5948c8e7fb69090ca4fa1ad2889df07ec5a8f36b234888209118bff17e1d0a8b WatchSource:0}: Error finding container 5948c8e7fb69090ca4fa1ad2889df07ec5a8f36b234888209118bff17e1d0a8b: Status 404 returned error can't find the container with id 5948c8e7fb69090ca4fa1ad2889df07ec5a8f36b234888209118bff17e1d0a8b
Sep 18 20:14:20.710264 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:20.710235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" event={"ID":"a107d23f-45bb-4867-aa13-99032c4c77db","Type":"ContainerStarted","Data":"c8f0a739d0b37f163020d4b738d73dbe4a4ab9a3560313c6adf748aef69cbe9f"}
Sep 18 20:14:20.710656 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:20.710292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" event={"ID":"a107d23f-45bb-4867-aa13-99032c4c77db","Type":"ContainerStarted","Data":"5948c8e7fb69090ca4fa1ad2889df07ec5a8f36b234888209118bff17e1d0a8b"}
Sep 18 20:14:22.719608 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:22.719525    2583 generic.go:356] "Generic (PLEG): container finished" podID="a107d23f-45bb-4867-aa13-99032c4c77db" containerID="c8f0a739d0b37f163020d4b738d73dbe4a4ab9a3560313c6adf748aef69cbe9f" exitCode=0
Sep 18 20:14:22.719608 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:22.719561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" event={"ID":"a107d23f-45bb-4867-aa13-99032c4c77db","Type":"ContainerDied","Data":"c8f0a739d0b37f163020d4b738d73dbe4a4ab9a3560313c6adf748aef69cbe9f"}
Sep 18 20:14:23.725824 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:23.725791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" event={"ID":"a107d23f-45bb-4867-aa13-99032c4c77db","Type":"ContainerStarted","Data":"9b0a849d3dea1d5e34d59555407fc9a2574771351ab58a8b4e0dc1ce2c7465e2"}
Sep 18 20:14:23.767402 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:23.767329    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" podStartSLOduration=2.690879312 podStartE2EDuration="4.767304183s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.076424871s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:23.764507997 +0000 UTC m=+2158.018714613" watchObservedRunningTime="2026-09-18 20:14:23.767304183 +0000 UTC m=+2158.021510787"
Sep 18 20:14:23.844459 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:23.844422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:29.845116 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:29.845080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:29.857287 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:29.857264    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:30.765247 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:30.765213    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:14:56.995487 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:14:56.995453    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"]
Sep 18 20:15:02.079634 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.079606    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"]
Sep 18 20:15:02.082950 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.082933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.091202 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.091177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-xtbjf\""
Sep 18 20:15:02.091310 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.091220    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 18 20:15:02.092843 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.092821    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 18 20:15:02.099609 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.099592    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 18 20:15:02.115658 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.115640    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"]
Sep 18 20:15:02.214042 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.214004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-dshm\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.214181 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.214062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.214181 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.214106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.214181 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.214150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.214313 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.214268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhcb\" (UniqueName: \"kubernetes.io/projected/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-kube-api-access-zjhcb\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.214375 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.214311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-home\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.314898 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.314868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-dshm\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.314898 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.314897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.315097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.314915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.315097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.315057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.315208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.315121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhcb\" (UniqueName: \"kubernetes.io/projected/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-kube-api-access-zjhcb\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.315208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.315175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-home\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.315389 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.315370    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.315513 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.315424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.315571 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.315514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-home\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.317424 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.317398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-dshm\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.317668 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.317651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.334073 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.334017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhcb\" (UniqueName: \"kubernetes.io/projected/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-kube-api-access-zjhcb\") pod \"tri-v1-kserve-79f8f98bc5-fmt8n\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.393016 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.392989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:02.721130 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:15:02.721101    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcba628_4cb7_4c26_b4a8_7493f71a3ef3.slice/crio-d5bd6955e0c350061a2e721ca6ace690eed6c59670595272fcc9d9a9ea0d0962 WatchSource:0}: Error finding container d5bd6955e0c350061a2e721ca6ace690eed6c59670595272fcc9d9a9ea0d0962: Status 404 returned error can't find the container with id d5bd6955e0c350061a2e721ca6ace690eed6c59670595272fcc9d9a9ea0d0962
Sep 18 20:15:02.723664 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.723636    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"]
Sep 18 20:15:02.885496 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.885463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n" event={"ID":"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3","Type":"ContainerStarted","Data":"d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641"}
Sep 18 20:15:02.885674 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:02.885512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n" event={"ID":"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3","Type":"ContainerStarted","Data":"d5bd6955e0c350061a2e721ca6ace690eed6c59670595272fcc9d9a9ea0d0962"}
Sep 18 20:15:03.393349 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:03.393320    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:12.014533 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.014492    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb" podUID="57f3d181-c53c-476c-b9a2-4ad00b104c19" containerName="main" containerID="cri-o://ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2" gracePeriod=45
Sep 18 20:15:12.252435 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.252405    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:15:12.285630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.285553    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n" podStartSLOduration=10.285536553 podStartE2EDuration="10.285536553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:02.93262883 +0000 UTC m=+2197.186835445" watchObservedRunningTime="2026-09-18 20:15:12.285536553 +0000 UTC m=+2206.539743157"
Sep 18 20:15:12.394135 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.394105    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:12.395563 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395540    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-dshm\") pod \"57f3d181-c53c-476c-b9a2-4ad00b104c19\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") "
Sep 18 20:15:12.395695 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395599    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-home\") pod \"57f3d181-c53c-476c-b9a2-4ad00b104c19\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") "
Sep 18 20:15:12.395695 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-tmp-dir\") pod \"57f3d181-c53c-476c-b9a2-4ad00b104c19\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") "
Sep 18 20:15:12.395695 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395674    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl7n5\" (UniqueName: \"kubernetes.io/projected/57f3d181-c53c-476c-b9a2-4ad00b104c19-kube-api-access-jl7n5\") pod \"57f3d181-c53c-476c-b9a2-4ad00b104c19\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") "
Sep 18 20:15:12.395865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395701    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-model-cache\") pod \"57f3d181-c53c-476c-b9a2-4ad00b104c19\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") "
Sep 18 20:15:12.395865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395732    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f3d181-c53c-476c-b9a2-4ad00b104c19-tls-certs\") pod \"57f3d181-c53c-476c-b9a2-4ad00b104c19\" (UID: \"57f3d181-c53c-476c-b9a2-4ad00b104c19\") "
Sep 18 20:15:12.395964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395861    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-home" (OuterVolumeSpecName: "home") pod "57f3d181-c53c-476c-b9a2-4ad00b104c19" (UID: "57f3d181-c53c-476c-b9a2-4ad00b104c19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:12.396021 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395957    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57f3d181-c53c-476c-b9a2-4ad00b104c19" (UID: "57f3d181-c53c-476c-b9a2-4ad00b104c19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:12.396021 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.395985    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:15:12.396090 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.396014    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-model-cache" (OuterVolumeSpecName: "model-cache") pod "57f3d181-c53c-476c-b9a2-4ad00b104c19" (UID: "57f3d181-c53c-476c-b9a2-4ad00b104c19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:12.397988 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.397965    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f3d181-c53c-476c-b9a2-4ad00b104c19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57f3d181-c53c-476c-b9a2-4ad00b104c19" (UID: "57f3d181-c53c-476c-b9a2-4ad00b104c19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:15:12.398101 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.397983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-dshm" (OuterVolumeSpecName: "dshm") pod "57f3d181-c53c-476c-b9a2-4ad00b104c19" (UID: "57f3d181-c53c-476c-b9a2-4ad00b104c19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:12.398101 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.398059    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f3d181-c53c-476c-b9a2-4ad00b104c19-kube-api-access-jl7n5" (OuterVolumeSpecName: "kube-api-access-jl7n5") pod "57f3d181-c53c-476c-b9a2-4ad00b104c19" (UID: "57f3d181-c53c-476c-b9a2-4ad00b104c19"). InnerVolumeSpecName "kube-api-access-jl7n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:12.406990 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.406973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:12.496765 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.496739    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:15:12.496883 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.496769    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7n5\" (UniqueName: \"kubernetes.io/projected/57f3d181-c53c-476c-b9a2-4ad00b104c19-kube-api-access-jl7n5\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:15:12.496883 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.496784    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:15:12.496883 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.496820    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f3d181-c53c-476c-b9a2-4ad00b104c19-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:15:12.496883 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.496831    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f3d181-c53c-476c-b9a2-4ad00b104c19-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:15:12.931672 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.931634    2583 generic.go:356] "Generic (PLEG): container finished" podID="57f3d181-c53c-476c-b9a2-4ad00b104c19" containerID="ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2" exitCode=0
Sep 18 20:15:12.931859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.931695    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"
Sep 18 20:15:12.931859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.931724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb" event={"ID":"57f3d181-c53c-476c-b9a2-4ad00b104c19","Type":"ContainerDied","Data":"ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2"}
Sep 18 20:15:12.931859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.931802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb" event={"ID":"57f3d181-c53c-476c-b9a2-4ad00b104c19","Type":"ContainerDied","Data":"2be2595d10a1f2537bd9531815573fa01da870b2089fe65a534a0288898165ec"}
Sep 18 20:15:12.931859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.931828    2583 scope.go:117] "RemoveContainer" containerID="ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2"
Sep 18 20:15:12.941504 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.941488    2583 scope.go:117] "RemoveContainer" containerID="ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2"
Sep 18 20:15:12.941795 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:15:12.941777    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2\": container with ID starting with ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2 not found: ID does not exist" containerID="ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2"
Sep 18 20:15:12.941850 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.941803    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2"} err="failed to get container status \"ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2\": rpc error: code = NotFound desc = could not find container \"ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2\": container with ID starting with ca9688f792fb40c2a0674e4a728c4d76235bf2fb53f1e05d63f96c5fd990f7a2 not found: ID does not exist"
Sep 18 20:15:12.944418 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.944403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:15:12.962223 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.962205    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"]
Sep 18 20:15:12.969861 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:12.969843    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n2hnb"]
Sep 18 20:15:13.965703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:13.965670    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f3d181-c53c-476c-b9a2-4ad00b104c19" path="/var/lib/kubelet/pods/57f3d181-c53c-476c-b9a2-4ad00b104c19/volumes"
Sep 18 20:15:35.941580 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.941543    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"]
Sep 18 20:15:35.942030 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.941935    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f3d181-c53c-476c-b9a2-4ad00b104c19" containerName="main"
Sep 18 20:15:35.942030 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.941948    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f3d181-c53c-476c-b9a2-4ad00b104c19" containerName="main"
Sep 18 20:15:35.942105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.942053    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f3d181-c53c-476c-b9a2-4ad00b104c19" containerName="main"
Sep 18 20:15:35.946282 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.946261    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:35.949446 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.949426    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 18 20:15:35.969183 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.969151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:35.969298 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.969245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-dshm\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:35.969298 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.969274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-home\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:35.969445 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.969369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:35.969445 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.969404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:35.969552 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.969458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btmw8\" (UniqueName: \"kubernetes.io/projected/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-kube-api-access-btmw8\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:35.970429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:35.970349    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"]
Sep 18 20:15:36.070479 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-dshm\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.070479 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-home\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmw8\" (UniqueName: \"kubernetes.io/projected/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-kube-api-access-btmw8\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.070972 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.071039 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.070978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.071039 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.071021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-home\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.072857 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.072838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-dshm\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.073123 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.073106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.084284 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.084261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmw8\" (UniqueName: \"kubernetes.io/projected/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-kube-api-access-btmw8\") pod \"tri-v2-kserve-dcb554bf5-rwv8h\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.255458 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.255388    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:36.537167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:36.537135    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"]
Sep 18 20:15:36.541317 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:15:36.541292    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62ff613_a4ea_4871_9cc4_eb10f36b52a1.slice/crio-c53fbb9f1304eaff766404fa06d9bfd57b920ed987c2a24fb5ef1bb2d546017c WatchSource:0}: Error finding container c53fbb9f1304eaff766404fa06d9bfd57b920ed987c2a24fb5ef1bb2d546017c: Status 404 returned error can't find the container with id c53fbb9f1304eaff766404fa06d9bfd57b920ed987c2a24fb5ef1bb2d546017c
Sep 18 20:15:37.034587 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:37.034546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h" event={"ID":"f62ff613-a4ea-4871-9cc4-eb10f36b52a1","Type":"ContainerStarted","Data":"deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636"}
Sep 18 20:15:37.034980 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:37.034611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h" event={"ID":"f62ff613-a4ea-4871-9cc4-eb10f36b52a1","Type":"ContainerStarted","Data":"c53fbb9f1304eaff766404fa06d9bfd57b920ed987c2a24fb5ef1bb2d546017c"}
Sep 18 20:15:37.078615 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:37.078563    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h" podStartSLOduration=2.078548088 podStartE2EDuration="2.078548088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:37.077888747 +0000 UTC m=+2231.332095351" watchObservedRunningTime="2026-09-18 20:15:37.078548088 +0000 UTC m=+2231.332754692"
Sep 18 20:15:37.255770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:37.255738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:46.255783 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:46.255749    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:46.268410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:46.268385    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:15:47.084213 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:15:47.084180    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:16:18.843829 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:18.843797    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"]
Sep 18 20:16:18.853211 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:16:18.853172    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:16:18.853211 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:16:18.853211 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:16:18.853211 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="9b0a849d3dea1d5e34d59555407fc9a2574771351ab58a8b4e0dc1ce2c7465e2" cmd=["/bin/sleep","15"]
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:18.853228    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" message="" err=<
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:16:18.853240    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" containerName="main" containerID="cri-o://9b0a849d3dea1d5e34d59555407fc9a2574771351ab58a8b4e0dc1ce2c7465e2"
Sep 18 20:16:18.853514 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:18.853286    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" containerName="main" containerID="cri-o://9b0a849d3dea1d5e34d59555407fc9a2574771351ab58a8b4e0dc1ce2c7465e2" gracePeriod=60
Sep 18 20:16:19.204603 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.204572    2583 generic.go:356] "Generic (PLEG): container finished" podID="a107d23f-45bb-4867-aa13-99032c4c77db" containerID="9b0a849d3dea1d5e34d59555407fc9a2574771351ab58a8b4e0dc1ce2c7465e2" exitCode=0
Sep 18 20:16:19.204738 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.204641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" event={"ID":"a107d23f-45bb-4867-aa13-99032c4c77db","Type":"ContainerDied","Data":"9b0a849d3dea1d5e34d59555407fc9a2574771351ab58a8b4e0dc1ce2c7465e2"}
Sep 18 20:16:19.204738 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.204700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968" event={"ID":"a107d23f-45bb-4867-aa13-99032c4c77db","Type":"ContainerDied","Data":"5948c8e7fb69090ca4fa1ad2889df07ec5a8f36b234888209118bff17e1d0a8b"}
Sep 18 20:16:19.204738 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.204718    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5948c8e7fb69090ca4fa1ad2889df07ec5a8f36b234888209118bff17e1d0a8b"
Sep 18 20:16:19.206844 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.206827    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:16:19.326918 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.326874    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-model-cache\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.326933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a107d23f-45bb-4867-aa13-99032c4c77db-cabundle-cert\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.326975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-kserve-provision-location\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.326995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmhb7\" (UniqueName: \"kubernetes.io/projected/a107d23f-45bb-4867-aa13-99032c4c77db-kube-api-access-hmhb7\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327017    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-home\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-tmp-dir\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327110    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-dshm\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a107d23f-45bb-4867-aa13-99032c4c77db-tls-certs\") pod \"a107d23f-45bb-4867-aa13-99032c4c77db\" (UID: \"a107d23f-45bb-4867-aa13-99032c4c77db\") "
Sep 18 20:16:19.327565 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327200    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-model-cache" (OuterVolumeSpecName: "model-cache") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.327565 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327396    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.327565 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.327565 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327453    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-home" (OuterVolumeSpecName: "home") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.327565 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.327487    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a107d23f-45bb-4867-aa13-99032c4c77db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:19.329780 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.329749    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a107d23f-45bb-4867-aa13-99032c4c77db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:19.330092 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.330069    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a107d23f-45bb-4867-aa13-99032c4c77db-kube-api-access-hmhb7" (OuterVolumeSpecName: "kube-api-access-hmhb7") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "kube-api-access-hmhb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:19.330202 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.330111    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-dshm" (OuterVolumeSpecName: "dshm") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.347584 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.347557    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a107d23f-45bb-4867-aa13-99032c4c77db" (UID: "a107d23f-45bb-4867-aa13-99032c4c77db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.428429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.428343    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.428429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.428386    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a107d23f-45bb-4867-aa13-99032c4c77db-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.428429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.428395    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a107d23f-45bb-4867-aa13-99032c4c77db-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.428429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.428403    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.428429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.428411    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhb7\" (UniqueName: \"kubernetes.io/projected/a107d23f-45bb-4867-aa13-99032c4c77db-kube-api-access-hmhb7\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.428429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.428419    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.428429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:19.428427    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a107d23f-45bb-4867-aa13-99032c4c77db-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:16:20.209511 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:20.209469    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"
Sep 18 20:16:20.228168 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:20.228088    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"]
Sep 18 20:16:20.229830 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:20.229806    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-xc968"]
Sep 18 20:16:21.966638 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:16:21.966607    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" path="/var/lib/kubelet/pods/a107d23f-45bb-4867-aa13-99032c4c77db/volumes"
Sep 18 20:17:15.410127 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:15.410097    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"]
Sep 18 20:17:19.419642 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:19.419608    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"]
Sep 18 20:17:30.429862 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.429813    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n" podUID="fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" containerName="main" containerID="cri-o://d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641" gracePeriod=45
Sep 18 20:17:30.665268 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.665245    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:17:30.781770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.781746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-home\") pod \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") "
Sep 18 20:17:30.781902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.781778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tmp-dir\") pod \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") "
Sep 18 20:17:30.781902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.781817    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjhcb\" (UniqueName: \"kubernetes.io/projected/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-kube-api-access-zjhcb\") pod \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") "
Sep 18 20:17:30.781902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.781853    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-dshm\") pod \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") "
Sep 18 20:17:30.781902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.781879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tls-certs\") pod \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") "
Sep 18 20:17:30.781902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.781903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-model-cache\") pod \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\" (UID: \"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3\") "
Sep 18 20:17:30.782161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.782033    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-home" (OuterVolumeSpecName: "home") pod "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" (UID: "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:30.782161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.782050    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" (UID: "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:30.782267 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.782206    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" (UID: "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:30.784227 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.784202    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-dshm" (OuterVolumeSpecName: "dshm") pod "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" (UID: "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:30.784327 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.784222    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" (UID: "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:30.784327 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.784205    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-kube-api-access-zjhcb" (OuterVolumeSpecName: "kube-api-access-zjhcb") pod "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" (UID: "fdcba628-4cb7-4c26-b4a8-7493f71a3ef3"). InnerVolumeSpecName "kube-api-access-zjhcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:30.882513 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.882488    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:30.882513 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.882512    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:30.882651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.882520    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:30.882651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.882548    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:30.882651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.882557    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhcb\" (UniqueName: \"kubernetes.io/projected/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-kube-api-access-zjhcb\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:30.882651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:30.882565    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:31.498990 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.498948    2583 generic.go:356] "Generic (PLEG): container finished" podID="fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" containerID="d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641" exitCode=0
Sep 18 20:17:31.498990 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.498996    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"
Sep 18 20:17:31.499495 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.499033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n" event={"ID":"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3","Type":"ContainerDied","Data":"d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641"}
Sep 18 20:17:31.499495 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.499085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n" event={"ID":"fdcba628-4cb7-4c26-b4a8-7493f71a3ef3","Type":"ContainerDied","Data":"d5bd6955e0c350061a2e721ca6ace690eed6c59670595272fcc9d9a9ea0d0962"}
Sep 18 20:17:31.499495 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.499112    2583 scope.go:117] "RemoveContainer" containerID="d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641"
Sep 18 20:17:31.507748 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.507733    2583 scope.go:117] "RemoveContainer" containerID="d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641"
Sep 18 20:17:31.508020 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:17:31.507999    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641\": container with ID starting with d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641 not found: ID does not exist" containerID="d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641"
Sep 18 20:17:31.508075 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.508028    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641"} err="failed to get container status \"d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641\": rpc error: code = NotFound desc = could not find container \"d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641\": container with ID starting with d3ad4e8a53b5dbe973f26b51b03adbbf1f8d4032b1f41fc3fa29a76f11d76641 not found: ID does not exist"
Sep 18 20:17:31.519166 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.519143    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"]
Sep 18 20:17:31.523460 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.523439    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-fmt8n"]
Sep 18 20:17:31.965835 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:31.965786    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" path="/var/lib/kubelet/pods/fdcba628-4cb7-4c26-b4a8-7493f71a3ef3/volumes"
Sep 18 20:17:34.438703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.438651    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h" podUID="f62ff613-a4ea-4871-9cc4-eb10f36b52a1" containerName="main" containerID="cri-o://deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636" gracePeriod=45
Sep 18 20:17:34.681674 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.681652    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:17:34.814220 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814196    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tls-certs\") pod \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") "
Sep 18 20:17:34.814416 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814253    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-model-cache\") pod \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") "
Sep 18 20:17:34.814416 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814293    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-dshm\") pod \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") "
Sep 18 20:17:34.814416 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814319    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btmw8\" (UniqueName: \"kubernetes.io/projected/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-kube-api-access-btmw8\") pod \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") "
Sep 18 20:17:34.814416 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814376    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tmp-dir\") pod \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") "
Sep 18 20:17:34.814416 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814411    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-home\") pod \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\" (UID: \"f62ff613-a4ea-4871-9cc4-eb10f36b52a1\") "
Sep 18 20:17:34.814696 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814557    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f62ff613-a4ea-4871-9cc4-eb10f36b52a1" (UID: "f62ff613-a4ea-4871-9cc4-eb10f36b52a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.814696 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814651    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f62ff613-a4ea-4871-9cc4-eb10f36b52a1" (UID: "f62ff613-a4ea-4871-9cc4-eb10f36b52a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.814696 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.814681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-home" (OuterVolumeSpecName: "home") pod "f62ff613-a4ea-4871-9cc4-eb10f36b52a1" (UID: "f62ff613-a4ea-4871-9cc4-eb10f36b52a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.816546 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.816519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f62ff613-a4ea-4871-9cc4-eb10f36b52a1" (UID: "f62ff613-a4ea-4871-9cc4-eb10f36b52a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:34.816655 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.816562    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-kube-api-access-btmw8" (OuterVolumeSpecName: "kube-api-access-btmw8") pod "f62ff613-a4ea-4871-9cc4-eb10f36b52a1" (UID: "f62ff613-a4ea-4871-9cc4-eb10f36b52a1"). InnerVolumeSpecName "kube-api-access-btmw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:34.816655 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.816592    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-dshm" (OuterVolumeSpecName: "dshm") pod "f62ff613-a4ea-4871-9cc4-eb10f36b52a1" (UID: "f62ff613-a4ea-4871-9cc4-eb10f36b52a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.915794 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.915773    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.915794 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.915792    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.915934 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.915801    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btmw8\" (UniqueName: \"kubernetes.io/projected/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-kube-api-access-btmw8\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.915934 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.915811    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.915934 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.915818    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.915934 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:34.915826    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ff613-a4ea-4871-9cc4-eb10f36b52a1-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:17:35.517991 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.517954    2583 generic.go:356] "Generic (PLEG): container finished" podID="f62ff613-a4ea-4871-9cc4-eb10f36b52a1" containerID="deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636" exitCode=0
Sep 18 20:17:35.518414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.517999    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"
Sep 18 20:17:35.518414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.518037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h" event={"ID":"f62ff613-a4ea-4871-9cc4-eb10f36b52a1","Type":"ContainerDied","Data":"deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636"}
Sep 18 20:17:35.518414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.518092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h" event={"ID":"f62ff613-a4ea-4871-9cc4-eb10f36b52a1","Type":"ContainerDied","Data":"c53fbb9f1304eaff766404fa06d9bfd57b920ed987c2a24fb5ef1bb2d546017c"}
Sep 18 20:17:35.518414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.518115    2583 scope.go:117] "RemoveContainer" containerID="deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636"
Sep 18 20:17:35.526458 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.526442    2583 scope.go:117] "RemoveContainer" containerID="deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636"
Sep 18 20:17:35.526750 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:17:35.526731    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636\": container with ID starting with deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636 not found: ID does not exist" containerID="deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636"
Sep 18 20:17:35.526808 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.526759    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636"} err="failed to get container status \"deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636\": rpc error: code = NotFound desc = could not find container \"deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636\": container with ID starting with deaafe3b3254e830afd7c2adbea80dcc1e652eedd53a6743bd68287ba4e52636 not found: ID does not exist"
Sep 18 20:17:35.538850 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.538827    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"]
Sep 18 20:17:35.543654 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.543634    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rwv8h"]
Sep 18 20:17:35.966083 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:35.965997    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f62ff613-a4ea-4871-9cc4-eb10f36b52a1" path="/var/lib/kubelet/pods/f62ff613-a4ea-4871-9cc4-eb10f36b52a1/volumes"
Sep 18 20:17:37.598649 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598620    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"]
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598934    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62ff613-a4ea-4871-9cc4-eb10f36b52a1" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598947    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62ff613-a4ea-4871-9cc4-eb10f36b52a1" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598970    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598976    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598982    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598987    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598994    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" containerName="storage-initializer"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.598999    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" containerName="storage-initializer"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.599047    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdcba628-4cb7-4c26-b4a8-7493f71a3ef3" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.599057    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a107d23f-45bb-4867-aa13-99032c4c77db" containerName="main"
Sep 18 20:17:37.599056 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.599063    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62ff613-a4ea-4871-9cc4-eb10f36b52a1" containerName="main"
Sep 18 20:17:37.603022 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.603001    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.605613 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.605591    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 18 20:17:37.606595 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.606578    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 18 20:17:37.606694 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.606605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 18 20:17:37.606902 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.606880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-mxffl\""
Sep 18 20:17:37.613504 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.613485    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"]
Sep 18 20:17:37.736684 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.736653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-home\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.736842 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.736690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-dshm\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.736842 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.736709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjb9\" (UniqueName: \"kubernetes.io/projected/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-kube-api-access-qxjb9\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.736842 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.736782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-model-cache\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.736983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.736876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.736983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.736898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tls-certs\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.837836 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.837811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-home\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.837850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-dshm\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.837875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjb9\" (UniqueName: \"kubernetes.io/projected/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-kube-api-access-qxjb9\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.837904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-model-cache\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.837970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.837996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tls-certs\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838296 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.838266    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-model-cache\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838379 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.838298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-home\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.838435 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.838384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.840412 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.840392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-dshm\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.840553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.840534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tls-certs\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.846786 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.846763    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjb9\" (UniqueName: \"kubernetes.io/projected/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-kube-api-access-qxjb9\") pod \"late-v1-kserve-56cbc8456c-7spf4\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:37.914190 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:37.914127    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:38.186945 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:38.186917    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"]
Sep 18 20:17:38.189328 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:17:38.189289    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a48476_ab6c_4f7f_b0bc_0b2ce4022d56.slice/crio-44bb50da238b12214a60797de48f12d10e3a98a110ce91ec985b33c2220c3e8e WatchSource:0}: Error finding container 44bb50da238b12214a60797de48f12d10e3a98a110ce91ec985b33c2220c3e8e: Status 404 returned error can't find the container with id 44bb50da238b12214a60797de48f12d10e3a98a110ce91ec985b33c2220c3e8e
Sep 18 20:17:38.539380 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:38.539330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4" event={"ID":"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56","Type":"ContainerStarted","Data":"50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503"}
Sep 18 20:17:38.539583 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:38.539413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4" event={"ID":"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56","Type":"ContainerStarted","Data":"44bb50da238b12214a60797de48f12d10e3a98a110ce91ec985b33c2220c3e8e"}
Sep 18 20:17:38.558327 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:38.558285    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4" podStartSLOduration=1.558272724 podStartE2EDuration="1.558272724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:38.557112676 +0000 UTC m=+2352.811319279" watchObservedRunningTime="2026-09-18 20:17:38.558272724 +0000 UTC m=+2352.812479348"
Sep 18 20:17:38.915105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:38.915025    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:47.914810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:47.914780    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:47.927273 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:47.927248    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:17:48.594237 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:17:48.594205    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:18:10.508558 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.508528    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"]
Sep 18 20:18:10.511914 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.511891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.514515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.514494    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 18 20:18:10.520278 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.520251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"]
Sep 18 20:18:10.576869 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.576842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-home\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.577026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.576873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-dshm\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.577026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.576900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkwc\" (UniqueName: \"kubernetes.io/projected/7d109c84-949e-4aa3-89f6-276925abb77d-kube-api-access-6dkwc\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.577026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.576927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-model-cache\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.577026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.576981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-tmp-dir\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.577026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.577012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d109c84-949e-4aa3-89f6-276925abb77d-tls-certs\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677392 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-home\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677573 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-dshm\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677573 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkwc\" (UniqueName: \"kubernetes.io/projected/7d109c84-949e-4aa3-89f6-276925abb77d-kube-api-access-6dkwc\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677573 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-model-cache\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677573 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-tmp-dir\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677573 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d109c84-949e-4aa3-89f6-276925abb77d-tls-certs\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677811    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-home\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-model-cache\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.677975 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.677908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-tmp-dir\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.679930 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.679908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-dshm\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.680210 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.680193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d109c84-949e-4aa3-89f6-276925abb77d-tls-certs\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.684927 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.684907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkwc\" (UniqueName: \"kubernetes.io/projected/7d109c84-949e-4aa3-89f6-276925abb77d-kube-api-access-6dkwc\") pod \"late-v2-kserve-86766546f-ncrd4\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:10.823027 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:10.822988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:11.096438 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:11.096407    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"]
Sep 18 20:18:11.099099 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:18:11.099076    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d109c84_949e_4aa3_89f6_276925abb77d.slice/crio-0ee3a0be3d4dc95639260f9851e0688956de7aa3be936f6809bf5f8d9fb79497 WatchSource:0}: Error finding container 0ee3a0be3d4dc95639260f9851e0688956de7aa3be936f6809bf5f8d9fb79497: Status 404 returned error can't find the container with id 0ee3a0be3d4dc95639260f9851e0688956de7aa3be936f6809bf5f8d9fb79497
Sep 18 20:18:11.681528 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:11.681492    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4" event={"ID":"7d109c84-949e-4aa3-89f6-276925abb77d","Type":"ContainerStarted","Data":"3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4"}
Sep 18 20:18:11.681915 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:11.681543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4" event={"ID":"7d109c84-949e-4aa3-89f6-276925abb77d","Type":"ContainerStarted","Data":"0ee3a0be3d4dc95639260f9851e0688956de7aa3be936f6809bf5f8d9fb79497"}
Sep 18 20:18:11.700117 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:11.700068    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4" podStartSLOduration=1.7000531909999999 podStartE2EDuration="1.700053191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:11.698570062 +0000 UTC m=+2385.952776666" watchObservedRunningTime="2026-09-18 20:18:11.700053191 +0000 UTC m=+2385.954259794"
Sep 18 20:18:11.824117 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:11.824087    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:20.823595 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:20.823562    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:20.836061 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:20.836036    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:21.736164 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:21.736137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:18:28.022831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:28.022801    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:18:28.029844 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:18:28.029826    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:19:10.640986 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:10.640952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"]
Sep 18 20:19:10.646159 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:10.646135    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"]
Sep 18 20:19:21.607914 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.607873    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"]
Sep 18 20:19:21.611396 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.611342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.615027 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.615001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 18 20:19:21.615027 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.615016    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 18 20:19:21.615227 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.615011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-lm22n\""
Sep 18 20:19:21.615227 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.615013    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 18 20:19:21.620265 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.620243    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"]
Sep 18 20:19:21.720532 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.720506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-dshm\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.720675 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.720542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.720675 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.720567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-home\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.720675 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.720584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b6fb3d-3bf8-439f-a59d-ae6735854524-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.720675 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.720634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqfp\" (UniqueName: \"kubernetes.io/projected/29b6fb3d-3bf8-439f-a59d-ae6735854524-kube-api-access-kbqfp\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.720833 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.720684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.821646 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.821618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b6fb3d-3bf8-439f-a59d-ae6735854524-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.821785 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.821654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqfp\" (UniqueName: \"kubernetes.io/projected/29b6fb3d-3bf8-439f-a59d-ae6735854524-kube-api-access-kbqfp\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.821785 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.821679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.821785 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.821740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-dshm\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.821785 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.821773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.821985 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.821812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-home\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.822187 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.822159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.822312 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.822204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.822312 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.822241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-home\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.824205 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.824182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-dshm\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.824611 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.824587    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b6fb3d-3bf8-439f-a59d-ae6735854524-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.829995 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.829975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqfp\" (UniqueName: \"kubernetes.io/projected/29b6fb3d-3bf8-439f-a59d-ae6735854524-kube-api-access-kbqfp\") pod \"delw-v1-kserve-6cf8cbb847-xgd6z\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:21.922014 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:21.921949    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:22.195284 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:22.195207    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"]
Sep 18 20:19:22.198183 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:19:22.198158    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b6fb3d_3bf8_439f_a59d_ae6735854524.slice/crio-1564954ae5791c3321fe45157651217799151392d8edf6792004d79353c4d33a WatchSource:0}: Error finding container 1564954ae5791c3321fe45157651217799151392d8edf6792004d79353c4d33a: Status 404 returned error can't find the container with id 1564954ae5791c3321fe45157651217799151392d8edf6792004d79353c4d33a
Sep 18 20:19:22.984969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:22.984931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z" event={"ID":"29b6fb3d-3bf8-439f-a59d-ae6735854524","Type":"ContainerStarted","Data":"92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e"}
Sep 18 20:19:22.985459 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:22.985033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z" event={"ID":"29b6fb3d-3bf8-439f-a59d-ae6735854524","Type":"ContainerStarted","Data":"1564954ae5791c3321fe45157651217799151392d8edf6792004d79353c4d33a"}
Sep 18 20:19:23.001198 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:23.001151    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z" podStartSLOduration=2.001136471 podStartE2EDuration="2.001136471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:22.999787863 +0000 UTC m=+2457.253994478" watchObservedRunningTime="2026-09-18 20:19:23.001136471 +0000 UTC m=+2457.255343076"
Sep 18 20:19:23.922735 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:23.922695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:25.660286 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.660244    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4" podUID="77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" containerName="main" containerID="cri-o://50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503" gracePeriod=45
Sep 18 20:19:25.664604 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.664542    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4" podUID="7d109c84-949e-4aa3-89f6-276925abb77d" containerName="main" containerID="cri-o://3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4" gracePeriod=45
Sep 18 20:19:25.725371 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.725323    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4" podUID="7d109c84-949e-4aa3-89f6-276925abb77d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 20:19:25.922376 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.922339    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:19:25.925098 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.925080    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:19:25.998015 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.997982    2583 generic.go:356] "Generic (PLEG): container finished" podID="77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" containerID="50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503" exitCode=0
Sep 18 20:19:25.998161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.998021    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"
Sep 18 20:19:25.998161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.998071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4" event={"ID":"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56","Type":"ContainerDied","Data":"50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503"}
Sep 18 20:19:25.998161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.998123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4" event={"ID":"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56","Type":"ContainerDied","Data":"44bb50da238b12214a60797de48f12d10e3a98a110ce91ec985b33c2220c3e8e"}
Sep 18 20:19:25.998161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.998152    2583 scope.go:117] "RemoveContainer" containerID="50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503"
Sep 18 20:19:25.999413 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.999390    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d109c84-949e-4aa3-89f6-276925abb77d" containerID="3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4" exitCode=0
Sep 18 20:19:25.999530 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.999419    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4" event={"ID":"7d109c84-949e-4aa3-89f6-276925abb77d","Type":"ContainerDied","Data":"3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4"}
Sep 18 20:19:25.999530 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.999433    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"
Sep 18 20:19:25.999530 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:25.999465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4" event={"ID":"7d109c84-949e-4aa3-89f6-276925abb77d","Type":"ContainerDied","Data":"0ee3a0be3d4dc95639260f9851e0688956de7aa3be936f6809bf5f8d9fb79497"}
Sep 18 20:19:26.006204 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.006186    2583 scope.go:117] "RemoveContainer" containerID="50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503"
Sep 18 20:19:26.006482 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:19:26.006462    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503\": container with ID starting with 50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503 not found: ID does not exist" containerID="50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503"
Sep 18 20:19:26.006556 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.006496    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503"} err="failed to get container status \"50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503\": rpc error: code = NotFound desc = could not find container \"50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503\": container with ID starting with 50db299a971cceb9dd752e059571d75c0db9e8223f363aa4f7c7d5758ac54503 not found: ID does not exist"
Sep 18 20:19:26.006556 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.006519    2583 scope.go:117] "RemoveContainer" containerID="3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4"
Sep 18 20:19:26.013761 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.013742    2583 scope.go:117] "RemoveContainer" containerID="3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4"
Sep 18 20:19:26.014020 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:19:26.014001    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4\": container with ID starting with 3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4 not found: ID does not exist" containerID="3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4"
Sep 18 20:19:26.014076 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.014028    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4"} err="failed to get container status \"3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4\": rpc error: code = NotFound desc = could not find container \"3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4\": container with ID starting with 3e4a05bc14a8dfcb8e6da89d5839ecd24b9ee134bd7b15c7ec80265b372f53e4 not found: ID does not exist"
Sep 18 20:19:26.055850 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.055823    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-home\") pod \"7d109c84-949e-4aa3-89f6-276925abb77d\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") "
Sep 18 20:19:26.055850 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.055854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tls-certs\") pod \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") "
Sep 18 20:19:26.056046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.055874    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-home\") pod \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") "
Sep 18 20:19:26.056046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.055908    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-tmp-dir\") pod \"7d109c84-949e-4aa3-89f6-276925abb77d\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") "
Sep 18 20:19:26.056046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.055922    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d109c84-949e-4aa3-89f6-276925abb77d-tls-certs\") pod \"7d109c84-949e-4aa3-89f6-276925abb77d\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") "
Sep 18 20:19:26.056046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.055941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dkwc\" (UniqueName: \"kubernetes.io/projected/7d109c84-949e-4aa3-89f6-276925abb77d-kube-api-access-6dkwc\") pod \"7d109c84-949e-4aa3-89f6-276925abb77d\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") "
Sep 18 20:19:26.056046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.055995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-dshm\") pod \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") "
Sep 18 20:19:26.056046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxjb9\" (UniqueName: \"kubernetes.io/projected/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-kube-api-access-qxjb9\") pod \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") "
Sep 18 20:19:26.056046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056037    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-dshm\") pod \"7d109c84-949e-4aa3-89f6-276925abb77d\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") "
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056067    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-model-cache\") pod \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") "
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tmp-dir\") pod \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\" (UID: \"77a48476-ab6c-4f7f-b0bc-0b2ce4022d56\") "
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-model-cache\") pod \"7d109c84-949e-4aa3-89f6-276925abb77d\" (UID: \"7d109c84-949e-4aa3-89f6-276925abb77d\") "
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056154    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-home" (OuterVolumeSpecName: "home") pod "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" (UID: "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056159    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-home" (OuterVolumeSpecName: "home") pod "7d109c84-949e-4aa3-89f6-276925abb77d" (UID: "7d109c84-949e-4aa3-89f6-276925abb77d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d109c84-949e-4aa3-89f6-276925abb77d" (UID: "7d109c84-949e-4aa3-89f6-276925abb77d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056397    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d109c84-949e-4aa3-89f6-276925abb77d" (UID: "7d109c84-949e-4aa3-89f6-276925abb77d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056407    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056425    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.056440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056437    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.057136 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056583    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-model-cache" (OuterVolumeSpecName: "model-cache") pod "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" (UID: "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.057136 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.056636    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" (UID: "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.058593 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.058570    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-kube-api-access-qxjb9" (OuterVolumeSpecName: "kube-api-access-qxjb9") pod "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" (UID: "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56"). InnerVolumeSpecName "kube-api-access-qxjb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:26.058593 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.058579    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d109c84-949e-4aa3-89f6-276925abb77d-kube-api-access-6dkwc" (OuterVolumeSpecName: "kube-api-access-6dkwc") pod "7d109c84-949e-4aa3-89f6-276925abb77d" (UID: "7d109c84-949e-4aa3-89f6-276925abb77d"). InnerVolumeSpecName "kube-api-access-6dkwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:26.058832 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.058816    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d109c84-949e-4aa3-89f6-276925abb77d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d109c84-949e-4aa3-89f6-276925abb77d" (UID: "7d109c84-949e-4aa3-89f6-276925abb77d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:19:26.058881 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.058837    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" (UID: "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:19:26.058979 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.058964    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-dshm" (OuterVolumeSpecName: "dshm") pod "7d109c84-949e-4aa3-89f6-276925abb77d" (UID: "7d109c84-949e-4aa3-89f6-276925abb77d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.059018 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.058962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-dshm" (OuterVolumeSpecName: "dshm") pod "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" (UID: "77a48476-ab6c-4f7f-b0bc-0b2ce4022d56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:26.157560 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157534    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxjb9\" (UniqueName: \"kubernetes.io/projected/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-kube-api-access-qxjb9\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157560 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157558    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157560 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157567    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157577    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157585    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d109c84-949e-4aa3-89f6-276925abb77d-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157593    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157600    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d109c84-949e-4aa3-89f6-276925abb77d-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157608    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dkwc\" (UniqueName: \"kubernetes.io/projected/7d109c84-949e-4aa3-89f6-276925abb77d-kube-api-access-6dkwc\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.157770 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.157614    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:19:26.324239 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.324208    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"]
Sep 18 20:19:26.326498 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.326464    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ncrd4"]
Sep 18 20:19:26.336946 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.336923    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"]
Sep 18 20:19:26.340704 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:26.340682    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-7spf4"]
Sep 18 20:19:27.966206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:27.966176    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" path="/var/lib/kubelet/pods/77a48476-ab6c-4f7f-b0bc-0b2ce4022d56/volumes"
Sep 18 20:19:27.966588 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:27.966544    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d109c84-949e-4aa3-89f6-276925abb77d" path="/var/lib/kubelet/pods/7d109c84-949e-4aa3-89f6-276925abb77d/volumes"
Sep 18 20:19:31.601032 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.600998    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"]
Sep 18 20:19:31.601430 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.601273    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" containerName="main"
Sep 18 20:19:31.601430 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.601283    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" containerName="main"
Sep 18 20:19:31.601430 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.601293    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d109c84-949e-4aa3-89f6-276925abb77d" containerName="main"
Sep 18 20:19:31.601430 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.601297    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d109c84-949e-4aa3-89f6-276925abb77d" containerName="main"
Sep 18 20:19:31.601430 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.601341    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a48476-ab6c-4f7f-b0bc-0b2ce4022d56" containerName="main"
Sep 18 20:19:31.601430 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.601349    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d109c84-949e-4aa3-89f6-276925abb77d" containerName="main"
Sep 18 20:19:31.605612 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.605595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.608065 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.608043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 18 20:19:31.615876 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.615852    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"]
Sep 18 20:19:31.699082 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.699049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.699250 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.699101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.699250 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.699198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-home\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.699250 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.699240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.699423 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.699269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.699423 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.699323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj95j\" (UniqueName: \"kubernetes.io/projected/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-kube-api-access-lj95j\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800226 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj95j\" (UniqueName: \"kubernetes.io/projected/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-kube-api-access-lj95j\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800415 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800415 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800557 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-home\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800557 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800557 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800704 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800669    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800793 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-home\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.800909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.800887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.802658 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.802638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.802932 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.802914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.808609 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.808567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj95j\" (UniqueName: \"kubernetes.io/projected/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-kube-api-access-lj95j\") pod \"delw-v2-kserve-5b84cc7ddf-26vg7\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.915254 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.915186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:31.922425 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.922402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:31.935854 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:31.935828    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:32.038469 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:32.038439    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:19:32.185506 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:32.185429    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"]
Sep 18 20:19:32.188239 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:19:32.188212    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee93bf2a_dbd8_490d_8ef7_ee359060fdb2.slice/crio-29f54a789ecf9113c7a47ed8bfa6105cdf9f8a9e18850465ead1188914b3d60d WatchSource:0}: Error finding container 29f54a789ecf9113c7a47ed8bfa6105cdf9f8a9e18850465ead1188914b3d60d: Status 404 returned error can't find the container with id 29f54a789ecf9113c7a47ed8bfa6105cdf9f8a9e18850465ead1188914b3d60d
Sep 18 20:19:33.033112 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:33.033078    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7" event={"ID":"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2","Type":"ContainerStarted","Data":"739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac"}
Sep 18 20:19:33.033545 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:33.033126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7" event={"ID":"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2","Type":"ContainerStarted","Data":"29f54a789ecf9113c7a47ed8bfa6105cdf9f8a9e18850465ead1188914b3d60d"}
Sep 18 20:19:33.051564 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:33.051517    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7" podStartSLOduration=2.051503775 podStartE2EDuration="2.051503775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:33.05010524 +0000 UTC m=+2467.304311844" watchObservedRunningTime="2026-09-18 20:19:33.051503775 +0000 UTC m=+2467.305710379"
Sep 18 20:19:33.915897 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:33.915866    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:41.916010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:41.915975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:41.928568 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:41.928544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:19:42.077789 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:19:42.077763    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:20:19.716705 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.716663    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"]
Sep 18 20:20:19.720338 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.720316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.723255 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.723235    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 20:20:19.724251 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.724218    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-f7bgh\""
Sep 18 20:20:19.724381 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.724251    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 20:20:19.724381 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.724279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 18 20:20:19.731686 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.731661    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"]
Sep 18 20:20:19.756326 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.756305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.756506 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.756332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6f4p\" (UniqueName: \"kubernetes.io/projected/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-kube-api-access-r6f4p\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.756506 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.756431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.756506 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.756489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.756700 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.756519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.756700 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.756548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.857716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.857685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.857855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.857726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.857855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.857752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.857855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.857778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.857855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.857828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.857855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.857853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6f4p\" (UniqueName: \"kubernetes.io/projected/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-kube-api-access-r6f4p\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.858160 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.858127    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.858273 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.858178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.858273 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.858213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.860768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.860743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.861145 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.861124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:19.868480 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:19.868456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6f4p\" (UniqueName: \"kubernetes.io/projected/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-kube-api-access-r6f4p\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:20.031505 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:20.031466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:20.302912 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:20.302877    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"]
Sep 18 20:20:20.305410 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:20:20.305382    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0353b6f2_df3a_4c73_98d8_b557e0f37e8d.slice/crio-3798a3ab7fb3dc53c8854889f4958b7e5de4b67f346fc742a2b7b70369c9a872 WatchSource:0}: Error finding container 3798a3ab7fb3dc53c8854889f4958b7e5de4b67f346fc742a2b7b70369c9a872: Status 404 returned error can't find the container with id 3798a3ab7fb3dc53c8854889f4958b7e5de4b67f346fc742a2b7b70369c9a872
Sep 18 20:20:21.229059 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:21.229022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" event={"ID":"0353b6f2-df3a-4c73-98d8-b557e0f37e8d","Type":"ContainerStarted","Data":"d50b9c58b117dd80ff71af8a5ca452a06e2b3af3e3441ff6475dcb73f9141549"}
Sep 18 20:20:21.229464 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:21.229085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" event={"ID":"0353b6f2-df3a-4c73-98d8-b557e0f37e8d","Type":"ContainerStarted","Data":"3798a3ab7fb3dc53c8854889f4958b7e5de4b67f346fc742a2b7b70369c9a872"}
Sep 18 20:20:21.248832 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:21.248785    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" podStartSLOduration=2.24877139 podStartE2EDuration="2.24877139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:21.247707756 +0000 UTC m=+2515.501914385" watchObservedRunningTime="2026-09-18 20:20:21.24877139 +0000 UTC m=+2515.502977995"
Sep 18 20:20:22.032012 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:22.031976    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:28.123249 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:28.123153    2583 scope.go:117] "RemoveContainer" containerID="9b0a849d3dea1d5e34d59555407fc9a2574771351ab58a8b4e0dc1ce2c7465e2"
Sep 18 20:20:28.139631 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:28.131556    2583 scope.go:117] "RemoveContainer" containerID="c8f0a739d0b37f163020d4b738d73dbe4a4ab9a3560313c6adf748aef69cbe9f"
Sep 18 20:20:30.031819 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:30.031791    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:30.044333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:30.044309    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:30.277646 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:30.277618    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:45.356397 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:45.356346    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"]
Sep 18 20:20:53.081933 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.081902    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"]
Sep 18 20:20:53.091107 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:20:53.091073    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:20:53.091107 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:20:53.091107 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:20:53.091107 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="d50b9c58b117dd80ff71af8a5ca452a06e2b3af3e3441ff6475dcb73f9141549" cmd=["/bin/sleep","15"]
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.091124    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" message="" err=<
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:20:53.091136    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" podUID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" containerName="main" containerID="cri-o://d50b9c58b117dd80ff71af8a5ca452a06e2b3af3e3441ff6475dcb73f9141549"
Sep 18 20:20:53.091439 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.091173    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" podUID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" containerName="main" containerID="cri-o://d50b9c58b117dd80ff71af8a5ca452a06e2b3af3e3441ff6475dcb73f9141549" gracePeriod=60
Sep 18 20:20:53.267886 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.267843    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" podUID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 20:20:53.363150 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.363102    2583 generic.go:356] "Generic (PLEG): container finished" podID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" containerID="d50b9c58b117dd80ff71af8a5ca452a06e2b3af3e3441ff6475dcb73f9141549" exitCode=0
Sep 18 20:20:53.363326 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.363186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" event={"ID":"0353b6f2-df3a-4c73-98d8-b557e0f37e8d","Type":"ContainerDied","Data":"d50b9c58b117dd80ff71af8a5ca452a06e2b3af3e3441ff6475dcb73f9141549"}
Sep 18 20:20:53.424518 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.424490    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:53.531372 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531332    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-home\") pod \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") "
Sep 18 20:20:53.531529 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531404    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-model-cache\") pod \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") "
Sep 18 20:20:53.531529 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531458    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tmp-dir\") pod \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") "
Sep 18 20:20:53.531529 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531487    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tls-certs\") pod \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") "
Sep 18 20:20:53.531700 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531552    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6f4p\" (UniqueName: \"kubernetes.io/projected/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-kube-api-access-r6f4p\") pod \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") "
Sep 18 20:20:53.531700 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531575    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-dshm\") pod \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\" (UID: \"0353b6f2-df3a-4c73-98d8-b557e0f37e8d\") "
Sep 18 20:20:53.531700 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-home" (OuterVolumeSpecName: "home") pod "0353b6f2-df3a-4c73-98d8-b557e0f37e8d" (UID: "0353b6f2-df3a-4c73-98d8-b557e0f37e8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:53.531816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531701    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0353b6f2-df3a-4c73-98d8-b557e0f37e8d" (UID: "0353b6f2-df3a-4c73-98d8-b557e0f37e8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:53.531816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531712    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0353b6f2-df3a-4c73-98d8-b557e0f37e8d" (UID: "0353b6f2-df3a-4c73-98d8-b557e0f37e8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:53.531816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531787    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:20:53.531816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531804    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:20:53.531943 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.531820    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:20:53.533809 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.533790    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-dshm" (OuterVolumeSpecName: "dshm") pod "0353b6f2-df3a-4c73-98d8-b557e0f37e8d" (UID: "0353b6f2-df3a-4c73-98d8-b557e0f37e8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:53.533869 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.533850    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0353b6f2-df3a-4c73-98d8-b557e0f37e8d" (UID: "0353b6f2-df3a-4c73-98d8-b557e0f37e8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:20:53.533905 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.533892    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-kube-api-access-r6f4p" (OuterVolumeSpecName: "kube-api-access-r6f4p") pod "0353b6f2-df3a-4c73-98d8-b557e0f37e8d" (UID: "0353b6f2-df3a-4c73-98d8-b557e0f37e8d"). InnerVolumeSpecName "kube-api-access-r6f4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:20:53.633016 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.632954    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6f4p\" (UniqueName: \"kubernetes.io/projected/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-kube-api-access-r6f4p\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:20:53.633016 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.632978    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:20:53.633016 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:53.632989    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0353b6f2-df3a-4c73-98d8-b557e0f37e8d-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:20:54.369453 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:54.369405    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6" event={"ID":"0353b6f2-df3a-4c73-98d8-b557e0f37e8d","Type":"ContainerDied","Data":"3798a3ab7fb3dc53c8854889f4958b7e5de4b67f346fc742a2b7b70369c9a872"}
Sep 18 20:20:54.369900 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:54.369470    2583 scope.go:117] "RemoveContainer" containerID="d50b9c58b117dd80ff71af8a5ca452a06e2b3af3e3441ff6475dcb73f9141549"
Sep 18 20:20:54.369900 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:54.369425    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"
Sep 18 20:20:54.394996 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:54.394964    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"]
Sep 18 20:20:54.404839 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:54.400341    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995qw9r6"]
Sep 18 20:20:55.966363 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:20:55.966318    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" path="/var/lib/kubelet/pods/0353b6f2-df3a-4c73-98d8-b557e0f37e8d/volumes"
Sep 18 20:21:00.377436 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.377395    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7" podUID="ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" containerName="main" containerID="cri-o://739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac" gracePeriod=45
Sep 18 20:21:00.627608 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.627552    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:21:00.785703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.785673    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-model-cache\") pod \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") "
Sep 18 20:21:00.785886 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.785715    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tls-certs\") pod \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") "
Sep 18 20:21:00.785886 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.785732    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj95j\" (UniqueName: \"kubernetes.io/projected/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-kube-api-access-lj95j\") pod \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") "
Sep 18 20:21:00.785886 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.785751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tmp-dir\") pod \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") "
Sep 18 20:21:00.785886 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.785804    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-dshm\") pod \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") "
Sep 18 20:21:00.785886 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.785822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-home\") pod \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\" (UID: \"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2\") "
Sep 18 20:21:00.786153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.785992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" (UID: "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:00.786153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.786027    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" (UID: "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:00.786153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.786127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-home" (OuterVolumeSpecName: "home") pod "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" (UID: "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:00.788216 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.788189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-dshm" (OuterVolumeSpecName: "dshm") pod "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" (UID: "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:00.788216 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.788201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-kube-api-access-lj95j" (OuterVolumeSpecName: "kube-api-access-lj95j") pod "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" (UID: "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2"). InnerVolumeSpecName "kube-api-access-lj95j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:00.788426 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.788270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" (UID: "ee93bf2a-dbd8-490d-8ef7-ee359060fdb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:00.887384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.887291    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:00.887384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.887315    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:00.887384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.887327    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:00.887384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.887339    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:00.887384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.887350    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj95j\" (UniqueName: \"kubernetes.io/projected/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-kube-api-access-lj95j\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:00.887630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:00.887412    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:01.402225 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.402191    2583 generic.go:356] "Generic (PLEG): container finished" podID="ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" containerID="739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac" exitCode=0
Sep 18 20:21:01.402729 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.402249    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"
Sep 18 20:21:01.402729 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.402254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7" event={"ID":"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2","Type":"ContainerDied","Data":"739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac"}
Sep 18 20:21:01.402729 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.402393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7" event={"ID":"ee93bf2a-dbd8-490d-8ef7-ee359060fdb2","Type":"ContainerDied","Data":"29f54a789ecf9113c7a47ed8bfa6105cdf9f8a9e18850465ead1188914b3d60d"}
Sep 18 20:21:01.402729 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.402417    2583 scope.go:117] "RemoveContainer" containerID="739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac"
Sep 18 20:21:01.411406 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.411382    2583 scope.go:117] "RemoveContainer" containerID="739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac"
Sep 18 20:21:01.411713 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:21:01.411694    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac\": container with ID starting with 739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac not found: ID does not exist" containerID="739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac"
Sep 18 20:21:01.411774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.411722    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac"} err="failed to get container status \"739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac\": rpc error: code = NotFound desc = could not find container \"739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac\": container with ID starting with 739fba773fc89c7f47861f57b83a5c3ff704dde172356fd31c7e80ad4b8d54ac not found: ID does not exist"
Sep 18 20:21:01.427792 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.427766    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"]
Sep 18 20:21:01.431928 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.431904    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-26vg7"]
Sep 18 20:21:01.804774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.804742    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"]
Sep 18 20:21:01.966941 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:01.966897    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" path="/var/lib/kubelet/pods/ee93bf2a-dbd8-490d-8ef7-ee359060fdb2/volumes"
Sep 18 20:21:16.823448 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:16.823395    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z" podUID="29b6fb3d-3bf8-439f-a59d-ae6735854524" containerName="main" containerID="cri-o://92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e" gracePeriod=45
Sep 18 20:21:17.063651 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.063623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:21:17.100925 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.100861    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-model-cache\") pod \"29b6fb3d-3bf8-439f-a59d-ae6735854524\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") "
Sep 18 20:21:17.100925 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.100906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-tmp-dir\") pod \"29b6fb3d-3bf8-439f-a59d-ae6735854524\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") "
Sep 18 20:21:17.101095 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.100931    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-home\") pod \"29b6fb3d-3bf8-439f-a59d-ae6735854524\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") "
Sep 18 20:21:17.101095 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.100945    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-dshm\") pod \"29b6fb3d-3bf8-439f-a59d-ae6735854524\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") "
Sep 18 20:21:17.101095 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.100981    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbqfp\" (UniqueName: \"kubernetes.io/projected/29b6fb3d-3bf8-439f-a59d-ae6735854524-kube-api-access-kbqfp\") pod \"29b6fb3d-3bf8-439f-a59d-ae6735854524\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") "
Sep 18 20:21:17.101095 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.101025    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b6fb3d-3bf8-439f-a59d-ae6735854524-tls-certs\") pod \"29b6fb3d-3bf8-439f-a59d-ae6735854524\" (UID: \"29b6fb3d-3bf8-439f-a59d-ae6735854524\") "
Sep 18 20:21:17.101258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.101163    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-home" (OuterVolumeSpecName: "home") pod "29b6fb3d-3bf8-439f-a59d-ae6735854524" (UID: "29b6fb3d-3bf8-439f-a59d-ae6735854524"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:17.101258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.101173    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-model-cache" (OuterVolumeSpecName: "model-cache") pod "29b6fb3d-3bf8-439f-a59d-ae6735854524" (UID: "29b6fb3d-3bf8-439f-a59d-ae6735854524"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:17.101258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.101193    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29b6fb3d-3bf8-439f-a59d-ae6735854524" (UID: "29b6fb3d-3bf8-439f-a59d-ae6735854524"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:17.103426 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.103392    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-dshm" (OuterVolumeSpecName: "dshm") pod "29b6fb3d-3bf8-439f-a59d-ae6735854524" (UID: "29b6fb3d-3bf8-439f-a59d-ae6735854524"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:17.103426 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.103410    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b6fb3d-3bf8-439f-a59d-ae6735854524-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29b6fb3d-3bf8-439f-a59d-ae6735854524" (UID: "29b6fb3d-3bf8-439f-a59d-ae6735854524"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:17.103570 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.103467    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b6fb3d-3bf8-439f-a59d-ae6735854524-kube-api-access-kbqfp" (OuterVolumeSpecName: "kube-api-access-kbqfp") pod "29b6fb3d-3bf8-439f-a59d-ae6735854524" (UID: "29b6fb3d-3bf8-439f-a59d-ae6735854524"). InnerVolumeSpecName "kube-api-access-kbqfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:17.202068 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.202038    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:17.202068 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.202066    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:17.202068 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.202074    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:17.202287 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.202080    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b6fb3d-3bf8-439f-a59d-ae6735854524-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:17.202287 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.202089    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbqfp\" (UniqueName: \"kubernetes.io/projected/29b6fb3d-3bf8-439f-a59d-ae6735854524-kube-api-access-kbqfp\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:17.202287 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.202098    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b6fb3d-3bf8-439f-a59d-ae6735854524-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:21:17.469078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.468992    2583 generic.go:356] "Generic (PLEG): container finished" podID="29b6fb3d-3bf8-439f-a59d-ae6735854524" containerID="92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e" exitCode=0
Sep 18 20:21:17.469078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.469031    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"
Sep 18 20:21:17.469078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.469034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z" event={"ID":"29b6fb3d-3bf8-439f-a59d-ae6735854524","Type":"ContainerDied","Data":"92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e"}
Sep 18 20:21:17.469330 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.469088    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z" event={"ID":"29b6fb3d-3bf8-439f-a59d-ae6735854524","Type":"ContainerDied","Data":"1564954ae5791c3321fe45157651217799151392d8edf6792004d79353c4d33a"}
Sep 18 20:21:17.469330 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.469117    2583 scope.go:117] "RemoveContainer" containerID="92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e"
Sep 18 20:21:17.477939 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.477922    2583 scope.go:117] "RemoveContainer" containerID="92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e"
Sep 18 20:21:17.478202 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:21:17.478184    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e\": container with ID starting with 92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e not found: ID does not exist" containerID="92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e"
Sep 18 20:21:17.478247 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.478209    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e"} err="failed to get container status \"92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e\": rpc error: code = NotFound desc = could not find container \"92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e\": container with ID starting with 92ea1bbb1a26ea0eb73dd2f5081a3fe8898de62b7cc0e687ad0a674656b7072e not found: ID does not exist"
Sep 18 20:21:17.495210 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.495187    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"]
Sep 18 20:21:17.500079 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.500058    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z"]
Sep 18 20:21:17.967010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:17.966977    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b6fb3d-3bf8-439f-a59d-ae6735854524" path="/var/lib/kubelet/pods/29b6fb3d-3bf8-439f-a59d-ae6735854524/volumes"
Sep 18 20:21:18.028336 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:18.028282    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-xgd6z" podUID="29b6fb3d-3bf8-439f-a59d-ae6735854524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded"
Sep 18 20:21:19.996178 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996148    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"]
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996413    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b6fb3d-3bf8-439f-a59d-ae6735854524" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996424    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b6fb3d-3bf8-439f-a59d-ae6735854524" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996433    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996438    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996449    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996454    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996493    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b6fb3d-3bf8-439f-a59d-ae6735854524" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996501    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee93bf2a-dbd8-490d-8ef7-ee359060fdb2" containerName="main"
Sep 18 20:21:19.996597 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:19.996508    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0353b6f2-df3a-4c73-98d8-b557e0f37e8d" containerName="main"
Sep 18 20:21:20.000429 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.000413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.004145 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.004116    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 18 20:21:20.004145 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.004140    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 18 20:21:20.004342 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.004147    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-bx94w\""
Sep 18 20:21:20.004342 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.004177    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 18 20:21:20.009287 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.009267    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"]
Sep 18 20:21:20.021492 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.021469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbgh\" (UniqueName: \"kubernetes.io/projected/27b9f4a5-dea5-419a-9852-0f09634b524f-kube-api-access-2fbgh\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.021605 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.021505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-home\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.021605 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.021526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b9f4a5-dea5-419a-9852-0f09634b524f-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.021605 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.021543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.021731 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.021633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.021731 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.021666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-dshm\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122683 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-dshm\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122683 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbgh\" (UniqueName: \"kubernetes.io/projected/27b9f4a5-dea5-419a-9852-0f09634b524f-kube-api-access-2fbgh\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122683 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-home\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122683 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b9f4a5-dea5-419a-9852-0f09634b524f-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122974 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122974 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.122974 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.122959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-home\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.123081 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.123015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.124932 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.124905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-dshm\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.125194 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.125177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b9f4a5-dea5-419a-9852-0f09634b524f-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.131327 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.131292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbgh\" (UniqueName: \"kubernetes.io/projected/27b9f4a5-dea5-419a-9852-0f09634b524f-kube-api-access-2fbgh\") pod \"roll-v1-kserve-69cf8bcfc-75rkk\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.310558 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.310533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:20.587210 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:20.587022    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"]
Sep 18 20:21:20.589760 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:21:20.589734    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b9f4a5_dea5_419a_9852_0f09634b524f.slice/crio-7889c4791fa6b866db255a86715e69e3fde424a19f83a15ed5be6603f8506940 WatchSource:0}: Error finding container 7889c4791fa6b866db255a86715e69e3fde424a19f83a15ed5be6603f8506940: Status 404 returned error can't find the container with id 7889c4791fa6b866db255a86715e69e3fde424a19f83a15ed5be6603f8506940
Sep 18 20:21:21.491048 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:21.491014    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk" event={"ID":"27b9f4a5-dea5-419a-9852-0f09634b524f","Type":"ContainerStarted","Data":"3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a"}
Sep 18 20:21:21.491569 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:21.491065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk" event={"ID":"27b9f4a5-dea5-419a-9852-0f09634b524f","Type":"ContainerStarted","Data":"7889c4791fa6b866db255a86715e69e3fde424a19f83a15ed5be6603f8506940"}
Sep 18 20:21:21.509610 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:21.509559    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk" podStartSLOduration=2.509544571 podStartE2EDuration="2.509544571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:21.508206495 +0000 UTC m=+2575.762413112" watchObservedRunningTime="2026-09-18 20:21:21.509544571 +0000 UTC m=+2575.763751184"
Sep 18 20:21:22.311188 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:22.311158    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:25.625213 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.625178    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"]
Sep 18 20:21:25.628204 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.628188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.631242 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.631223    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 18 20:21:25.647939 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.647917    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"]
Sep 18 20:21:25.663463 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.663442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwf8p\" (UniqueName: \"kubernetes.io/projected/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-kube-api-access-cwf8p\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.663564 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.663482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.663564 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.663512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-dshm\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.663564 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.663552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.663702 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.663567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-home\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.663702 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.663583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764285 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwf8p\" (UniqueName: \"kubernetes.io/projected/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-kube-api-access-cwf8p\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764433 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764433 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-dshm\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764433 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764433 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-home\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764654 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764760 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764850 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764814    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-home\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.764896 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.764870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.766733 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.766717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-dshm\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.767041 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.767025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.774340 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.774319    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwf8p\" (UniqueName: \"kubernetes.io/projected/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-kube-api-access-cwf8p\") pod \"roll-v2-kserve-d4dbf9d67-gq87f\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:25.938767 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:25.938674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:26.219994 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:26.219962    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"]
Sep 18 20:21:26.220261 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:21:26.220238    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bb98b7_b3c2_4706_8eb9_8d20e3c6c000.slice/crio-d604a6482999c9c7a30de72485b346258a4deb3e893551def0e87ba2952e2ba0 WatchSource:0}: Error finding container d604a6482999c9c7a30de72485b346258a4deb3e893551def0e87ba2952e2ba0: Status 404 returned error can't find the container with id d604a6482999c9c7a30de72485b346258a4deb3e893551def0e87ba2952e2ba0
Sep 18 20:21:26.512559 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:26.512522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f" event={"ID":"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000","Type":"ContainerStarted","Data":"00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9"}
Sep 18 20:21:26.512739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:26.512585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f" event={"ID":"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000","Type":"ContainerStarted","Data":"d604a6482999c9c7a30de72485b346258a4deb3e893551def0e87ba2952e2ba0"}
Sep 18 20:21:26.533510 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:26.533461    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f" podStartSLOduration=1.533446447 podStartE2EDuration="1.533446447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:26.532436195 +0000 UTC m=+2580.786642800" watchObservedRunningTime="2026-09-18 20:21:26.533446447 +0000 UTC m=+2580.787653050"
Sep 18 20:21:26.939710 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:26.939629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:30.311567 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:30.311538    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:30.324506 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:30.324485    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:30.539406 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:30.539380    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:21:35.939803 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:35.939773    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:35.952269 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:35.952238    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:21:36.564090 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:21:36.564061    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:22:31.821041 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:31.821007    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"]
Sep 18 20:22:32.819950 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:32.819920    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"]
Sep 18 20:22:46.840613 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:46.840559    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk" podUID="27b9f4a5-dea5-419a-9852-0f09634b524f" containerName="main" containerID="cri-o://3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a" gracePeriod=45
Sep 18 20:22:47.082941 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.082918    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:22:47.182472 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182405    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-dshm\") pod \"27b9f4a5-dea5-419a-9852-0f09634b524f\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") "
Sep 18 20:22:47.182594 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182483    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fbgh\" (UniqueName: \"kubernetes.io/projected/27b9f4a5-dea5-419a-9852-0f09634b524f-kube-api-access-2fbgh\") pod \"27b9f4a5-dea5-419a-9852-0f09634b524f\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") "
Sep 18 20:22:47.182594 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182517    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-home\") pod \"27b9f4a5-dea5-419a-9852-0f09634b524f\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") "
Sep 18 20:22:47.182594 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-tmp-dir\") pod \"27b9f4a5-dea5-419a-9852-0f09634b524f\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") "
Sep 18 20:22:47.182594 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b9f4a5-dea5-419a-9852-0f09634b524f-tls-certs\") pod \"27b9f4a5-dea5-419a-9852-0f09634b524f\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") "
Sep 18 20:22:47.182797 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182603    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-model-cache\") pod \"27b9f4a5-dea5-419a-9852-0f09634b524f\" (UID: \"27b9f4a5-dea5-419a-9852-0f09634b524f\") "
Sep 18 20:22:47.182855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182804    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-home" (OuterVolumeSpecName: "home") pod "27b9f4a5-dea5-419a-9852-0f09634b524f" (UID: "27b9f4a5-dea5-419a-9852-0f09634b524f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:47.182855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182818    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27b9f4a5-dea5-419a-9852-0f09634b524f" (UID: "27b9f4a5-dea5-419a-9852-0f09634b524f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:47.182976 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.182925    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-model-cache" (OuterVolumeSpecName: "model-cache") pod "27b9f4a5-dea5-419a-9852-0f09634b524f" (UID: "27b9f4a5-dea5-419a-9852-0f09634b524f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:47.184768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.184727    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-dshm" (OuterVolumeSpecName: "dshm") pod "27b9f4a5-dea5-419a-9852-0f09634b524f" (UID: "27b9f4a5-dea5-419a-9852-0f09634b524f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:47.184875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.184776    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b9f4a5-dea5-419a-9852-0f09634b524f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27b9f4a5-dea5-419a-9852-0f09634b524f" (UID: "27b9f4a5-dea5-419a-9852-0f09634b524f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:47.184875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.184796    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b9f4a5-dea5-419a-9852-0f09634b524f-kube-api-access-2fbgh" (OuterVolumeSpecName: "kube-api-access-2fbgh") pod "27b9f4a5-dea5-419a-9852-0f09634b524f" (UID: "27b9f4a5-dea5-419a-9852-0f09634b524f"). InnerVolumeSpecName "kube-api-access-2fbgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:47.283981 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.283955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fbgh\" (UniqueName: \"kubernetes.io/projected/27b9f4a5-dea5-419a-9852-0f09634b524f-kube-api-access-2fbgh\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:47.283981 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.283976    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:47.283981 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.283985    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:47.284167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.283994    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b9f4a5-dea5-419a-9852-0f09634b524f-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:47.284167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.284001    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:47.284167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.284010    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b9f4a5-dea5-419a-9852-0f09634b524f-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:47.838678 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.838623    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f" podUID="97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" containerName="main" containerID="cri-o://00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9" gracePeriod=45
Sep 18 20:22:47.845556 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.845518    2583 generic.go:356] "Generic (PLEG): container finished" podID="27b9f4a5-dea5-419a-9852-0f09634b524f" containerID="3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a" exitCode=0
Sep 18 20:22:47.845924 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.845570    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"
Sep 18 20:22:47.845924 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.845570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk" event={"ID":"27b9f4a5-dea5-419a-9852-0f09634b524f","Type":"ContainerDied","Data":"3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a"}
Sep 18 20:22:47.845924 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.845633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk" event={"ID":"27b9f4a5-dea5-419a-9852-0f09634b524f","Type":"ContainerDied","Data":"7889c4791fa6b866db255a86715e69e3fde424a19f83a15ed5be6603f8506940"}
Sep 18 20:22:47.845924 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.845668    2583 scope.go:117] "RemoveContainer" containerID="3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a"
Sep 18 20:22:47.914822 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.914796    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"]
Sep 18 20:22:47.918581 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.918555    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-75rkk"]
Sep 18 20:22:47.948478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.948457    2583 scope.go:117] "RemoveContainer" containerID="3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a"
Sep 18 20:22:47.948791 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:22:47.948774    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a\": container with ID starting with 3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a not found: ID does not exist" containerID="3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a"
Sep 18 20:22:47.948841 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.948814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a"} err="failed to get container status \"3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a\": rpc error: code = NotFound desc = could not find container \"3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a\": container with ID starting with 3afd8fc4972d353e406565a87da2889a76afe326fc778f81760035a89ec0f46a not found: ID does not exist"
Sep 18 20:22:47.968256 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:47.968223    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b9f4a5-dea5-419a-9852-0f09634b524f" path="/var/lib/kubelet/pods/27b9f4a5-dea5-419a-9852-0f09634b524f/volumes"
Sep 18 20:22:48.075881 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.075854    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:22:48.190256 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwf8p\" (UniqueName: \"kubernetes.io/projected/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-kube-api-access-cwf8p\") pod \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") "
Sep 18 20:22:48.190256 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190209    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tls-certs\") pod \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") "
Sep 18 20:22:48.190256 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tmp-dir\") pod \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") "
Sep 18 20:22:48.190256 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190259    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-dshm\") pod \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") "
Sep 18 20:22:48.190645 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190324    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-home\") pod \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") "
Sep 18 20:22:48.190645 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-model-cache\") pod \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\" (UID: \"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000\") "
Sep 18 20:22:48.190645 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190590    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" (UID: "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:48.190645 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190614    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-home" (OuterVolumeSpecName: "home") pod "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" (UID: "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:48.190823 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190707    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:48.190823 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190725    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:48.190823 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.190727    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-model-cache" (OuterVolumeSpecName: "model-cache") pod "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" (UID: "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:48.192631 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.192605    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-dshm" (OuterVolumeSpecName: "dshm") pod "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" (UID: "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:48.192748 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.192667    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-kube-api-access-cwf8p" (OuterVolumeSpecName: "kube-api-access-cwf8p") pod "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" (UID: "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000"). InnerVolumeSpecName "kube-api-access-cwf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:48.192748 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.192726    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" (UID: "97bb98b7-b3c2-4706-8eb9-8d20e3c6c000"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:48.291491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.291458    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:48.291491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.291483    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:48.291491 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.291495    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwf8p\" (UniqueName: \"kubernetes.io/projected/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-kube-api-access-cwf8p\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:48.291707 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.291504    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:22:48.851581 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.851547    2583 generic.go:356] "Generic (PLEG): container finished" podID="97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" containerID="00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9" exitCode=0
Sep 18 20:22:48.852077 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.851614    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"
Sep 18 20:22:48.852077 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.851649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f" event={"ID":"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000","Type":"ContainerDied","Data":"00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9"}
Sep 18 20:22:48.852077 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.851710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f" event={"ID":"97bb98b7-b3c2-4706-8eb9-8d20e3c6c000","Type":"ContainerDied","Data":"d604a6482999c9c7a30de72485b346258a4deb3e893551def0e87ba2952e2ba0"}
Sep 18 20:22:48.852077 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.851745    2583 scope.go:117] "RemoveContainer" containerID="00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9"
Sep 18 20:22:48.860403 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.860385    2583 scope.go:117] "RemoveContainer" containerID="00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9"
Sep 18 20:22:48.860660 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:22:48.860644    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9\": container with ID starting with 00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9 not found: ID does not exist" containerID="00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9"
Sep 18 20:22:48.860705 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.860667    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9"} err="failed to get container status \"00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9\": rpc error: code = NotFound desc = could not find container \"00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9\": container with ID starting with 00553a16cad6c06b023305ccf08a93de16457ac9c7920b943a7ac2781d7a26b9 not found: ID does not exist"
Sep 18 20:22:48.875010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.874989    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"]
Sep 18 20:22:48.881202 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:48.881184    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-gq87f"]
Sep 18 20:22:49.965684 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:49.965651    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" path="/var/lib/kubelet/pods/97bb98b7-b3c2-4706-8eb9-8d20e3c6c000/volumes"
Sep 18 20:22:51.865790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.865759    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"]
Sep 18 20:22:51.866153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.866025    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b9f4a5-dea5-419a-9852-0f09634b524f" containerName="main"
Sep 18 20:22:51.866153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.866035    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b9f4a5-dea5-419a-9852-0f09634b524f" containerName="main"
Sep 18 20:22:51.866153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.866046    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" containerName="main"
Sep 18 20:22:51.866153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.866051    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" containerName="main"
Sep 18 20:22:51.866153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.866099    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b9f4a5-dea5-419a-9852-0f09634b524f" containerName="main"
Sep 18 20:22:51.866153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.866108    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bb98b7-b3c2-4706-8eb9-8d20e3c6c000" containerName="main"
Sep 18 20:22:51.870366 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.870335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:51.875333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.875314    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-hd7fv\""
Sep 18 20:22:51.875772 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.875745    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 18 20:22:51.875870 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.875806    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 18 20:22:51.875870 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.875748    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 18 20:22:51.886805 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.886782    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"]
Sep 18 20:22:51.918922 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.918886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf211e-fe51-4f05-871f-69e92475fac1-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:51.919031 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.918928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-model-cache\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:51.919031 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.918946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smn9\" (UniqueName: \"kubernetes.io/projected/6faf211e-fe51-4f05-871f-69e92475fac1-kube-api-access-5smn9\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:51.919031 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.918973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-home\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:51.919197 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.919049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:51.919197 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:51.919107    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-dshm\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020250 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-home\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-dshm\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020518 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf211e-fe51-4f05-871f-69e92475fac1-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020518 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-model-cache\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020518 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smn9\" (UniqueName: \"kubernetes.io/projected/6faf211e-fe51-4f05-871f-69e92475fac1-kube-api-access-5smn9\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020699 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-home\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020759 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.020846 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.020826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-model-cache\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.022810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.022780    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-dshm\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.023163 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.023145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf211e-fe51-4f05-871f-69e92475fac1-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.033591 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.033569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smn9\" (UniqueName: \"kubernetes.io/projected/6faf211e-fe51-4f05-871f-69e92475fac1-kube-api-access-5smn9\") pod \"owner-v2-kserve-5cf8959f95-tmsh2\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.180868 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.180800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:22:52.454751 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.454726    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"]
Sep 18 20:22:52.457482 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:22:52.457449    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faf211e_fe51_4f05_871f_69e92475fac1.slice/crio-0d5f58a3021a2db107068f03f5770f784e7c79d59ec8da5850d95e67ab7ac2a9 WatchSource:0}: Error finding container 0d5f58a3021a2db107068f03f5770f784e7c79d59ec8da5850d95e67ab7ac2a9: Status 404 returned error can't find the container with id 0d5f58a3021a2db107068f03f5770f784e7c79d59ec8da5850d95e67ab7ac2a9
Sep 18 20:22:52.872750 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.872715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2" event={"ID":"6faf211e-fe51-4f05-871f-69e92475fac1","Type":"ContainerStarted","Data":"c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342"}
Sep 18 20:22:52.873143 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.872765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2" event={"ID":"6faf211e-fe51-4f05-871f-69e92475fac1","Type":"ContainerStarted","Data":"0d5f58a3021a2db107068f03f5770f784e7c79d59ec8da5850d95e67ab7ac2a9"}
Sep 18 20:22:52.890301 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:52.890255    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2" podStartSLOduration=1.8902405199999999 podStartE2EDuration="1.89024052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:52.888428835 +0000 UTC m=+2667.142635439" watchObservedRunningTime="2026-09-18 20:22:52.89024052 +0000 UTC m=+2667.144447124"
Sep 18 20:22:53.181926 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:22:53.181851    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:23:02.181433 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:02.181388    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:23:02.194715 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:02.194691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:23:02.925724 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:02.925695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:23:28.046568 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:28.046537    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:23:28.054301 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:28.054280    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:23:38.794092 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.794059    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"]
Sep 18 20:23:38.798199 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.798176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:38.801615 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.801592    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 20:23:38.801721 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.801632    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ckdwr\""
Sep 18 20:23:38.801721 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.801650    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 20:23:38.801721 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.801633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 20:23:38.808753 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.808728    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"]
Sep 18 20:23:38.970111 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.970077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18856a-9aca-4de6-bb5b-bddd57029bd2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:38.970278 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.970126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:38.970278 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.970145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:38.970278 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.970161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5x4\" (UniqueName: \"kubernetes.io/projected/db18856a-9aca-4de6-bb5b-bddd57029bd2-kube-api-access-qp5x4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:38.970278 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.970227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:38.970278 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:38.970260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.070632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18856a-9aca-4de6-bb5b-bddd57029bd2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.070679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.070699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.070716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.070715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5x4\" (UniqueName: \"kubernetes.io/projected/db18856a-9aca-4de6-bb5b-bddd57029bd2-kube-api-access-qp5x4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.071055 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.070739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.071055 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.070770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.071171 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.071142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.071249 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.071174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.071249 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.071204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.073386 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.073342    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.073519 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.073444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18856a-9aca-4de6-bb5b-bddd57029bd2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.083840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.083818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5x4\" (UniqueName: \"kubernetes.io/projected/db18856a-9aca-4de6-bb5b-bddd57029bd2-kube-api-access-qp5x4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.109998 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.109967    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:39.395135 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:39.395060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"]
Sep 18 20:23:39.403286 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:23:39.403259    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb18856a_9aca_4de6_bb5b_bddd57029bd2.slice/crio-6aacc08d9cc569c24f41ca9e9a8aba50cecd7741cecdee101c133b51e4d360f1 WatchSource:0}: Error finding container 6aacc08d9cc569c24f41ca9e9a8aba50cecd7741cecdee101c133b51e4d360f1: Status 404 returned error can't find the container with id 6aacc08d9cc569c24f41ca9e9a8aba50cecd7741cecdee101c133b51e4d360f1
Sep 18 20:23:40.064424 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:40.064385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" event={"ID":"db18856a-9aca-4de6-bb5b-bddd57029bd2","Type":"ContainerStarted","Data":"be9d52890ed5b7f453661a54e13259c11e02476c9027d7c78890b0c3b5ae481d"}
Sep 18 20:23:40.064909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:40.064443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" event={"ID":"db18856a-9aca-4de6-bb5b-bddd57029bd2","Type":"ContainerStarted","Data":"6aacc08d9cc569c24f41ca9e9a8aba50cecd7741cecdee101c133b51e4d360f1"}
Sep 18 20:23:40.091547 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:40.091499    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" podStartSLOduration=2.091483889 podStartE2EDuration="2.091483889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:40.089739462 +0000 UTC m=+2714.343946078" watchObservedRunningTime="2026-09-18 20:23:40.091483889 +0000 UTC m=+2714.345690493"
Sep 18 20:23:40.110952 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:40.110927    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:49.110951 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:49.110909    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:49.123304 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:49.123278    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:23:50.115192 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:23:50.115157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:24:05.045420 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.045392    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"]
Sep 18 20:24:05.048673 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.048657    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.051940 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.051916    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-6s42j\""
Sep 18 20:24:05.052061 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.052019    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 18 20:24:05.052275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.052252    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 18 20:24:05.052910 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.052894    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 18 20:24:05.060709 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.060805 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.060805 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.060805 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d059d02-fd40-4515-9356-bc207f394dd0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.060921 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.060921 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.060921 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkfb\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-kube-api-access-zlkfb\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.061020 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.060956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.061020 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.061005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d059d02-fd40-4515-9356-bc207f394dd0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.131032 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.130991    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"]
Sep 18 20:24:05.161540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d059d02-fd40-4515-9356-bc207f394dd0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161670 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161670 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161670 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkfb\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-kube-api-access-zlkfb\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161670 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161891 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d059d02-fd40-4515-9356-bc207f394dd0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161891 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161891 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.161891 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.161799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.162104 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.162043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.162164 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.162143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.162225 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.162211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.162397 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.162378    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.162912 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.162887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d059d02-fd40-4515-9356-bc207f394dd0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.164265 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.164246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.164534 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.164514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d059d02-fd40-4515-9356-bc207f394dd0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.180895 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.180869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.182602 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.182583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkfb\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-kube-api-access-zlkfb\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-sw2xd\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.361682 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.361602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:05.649254 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.649227    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"]
Sep 18 20:24:05.651233 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:24:05.651200    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d059d02_fd40_4515_9356_bc207f394dd0.slice/crio-eb6b4e4cc44fff9d491e78cf1015d06082893c44a0da5d33facf261042bfc6e4 WatchSource:0}: Error finding container eb6b4e4cc44fff9d491e78cf1015d06082893c44a0da5d33facf261042bfc6e4: Status 404 returned error can't find the container with id eb6b4e4cc44fff9d491e78cf1015d06082893c44a0da5d33facf261042bfc6e4
Sep 18 20:24:05.653200 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:05.653184    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:24:06.171761 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:06.171656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" event={"ID":"8d059d02-fd40-4515-9356-bc207f394dd0","Type":"ContainerStarted","Data":"eb6b4e4cc44fff9d491e78cf1015d06082893c44a0da5d33facf261042bfc6e4"}
Sep 18 20:24:07.969703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:07.969669    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:24:07.969969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:07.969741    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:24:07.969969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:07.969776    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:24:08.183472 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:08.183435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" event={"ID":"8d059d02-fd40-4515-9356-bc207f394dd0","Type":"ContainerStarted","Data":"97b8998f3242f47efb3fbac95cede838b43ae3559ebc0a10a7a1d0a8623aa8dc"}
Sep 18 20:24:08.205904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:08.205849    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" podStartSLOduration=0.889672883 podStartE2EDuration="3.205832286s" totalImagesPullingTime="2.316159403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:08.204652323 +0000 UTC m=+2742.458858928" watchObservedRunningTime="2026-09-18 20:24:08.205832286 +0000 UTC m=+2742.460038889"
Sep 18 20:24:08.362130 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:08.362100    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:09.366905 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:09.366863    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:09.768921 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:09.768889    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"]
Sep 18 20:24:10.192201 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:10.192099    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:10.193064 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:10.193041    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:24.788446 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:24.788391    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2" podUID="6faf211e-fe51-4f05-871f-69e92475fac1" containerName="main" containerID="cri-o://c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342" gracePeriod=45
Sep 18 20:24:24.915644 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:24.915610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2" podUID="6faf211e-fe51-4f05-871f-69e92475fac1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 18 20:24:25.029163 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.029141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:24:25.117253 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117176    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-model-cache\") pod \"6faf211e-fe51-4f05-871f-69e92475fac1\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") "
Sep 18 20:24:25.117253 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117225    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-tmp-dir\") pod \"6faf211e-fe51-4f05-871f-69e92475fac1\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") "
Sep 18 20:24:25.117253 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-dshm\") pod \"6faf211e-fe51-4f05-871f-69e92475fac1\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") "
Sep 18 20:24:25.117555 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117264    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5smn9\" (UniqueName: \"kubernetes.io/projected/6faf211e-fe51-4f05-871f-69e92475fac1-kube-api-access-5smn9\") pod \"6faf211e-fe51-4f05-871f-69e92475fac1\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") "
Sep 18 20:24:25.117555 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117306    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-home\") pod \"6faf211e-fe51-4f05-871f-69e92475fac1\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") "
Sep 18 20:24:25.117555 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117338    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf211e-fe51-4f05-871f-69e92475fac1-tls-certs\") pod \"6faf211e-fe51-4f05-871f-69e92475fac1\" (UID: \"6faf211e-fe51-4f05-871f-69e92475fac1\") "
Sep 18 20:24:25.117555 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-model-cache" (OuterVolumeSpecName: "model-cache") pod "6faf211e-fe51-4f05-871f-69e92475fac1" (UID: "6faf211e-fe51-4f05-871f-69e92475fac1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:25.117734 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117551    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6faf211e-fe51-4f05-871f-69e92475fac1" (UID: "6faf211e-fe51-4f05-871f-69e92475fac1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:25.117734 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.117569    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-home" (OuterVolumeSpecName: "home") pod "6faf211e-fe51-4f05-871f-69e92475fac1" (UID: "6faf211e-fe51-4f05-871f-69e92475fac1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:25.119666 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.119644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-dshm" (OuterVolumeSpecName: "dshm") pod "6faf211e-fe51-4f05-871f-69e92475fac1" (UID: "6faf211e-fe51-4f05-871f-69e92475fac1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:25.119666 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.119652    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6faf211e-fe51-4f05-871f-69e92475fac1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6faf211e-fe51-4f05-871f-69e92475fac1" (UID: "6faf211e-fe51-4f05-871f-69e92475fac1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:25.119787 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.119672    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faf211e-fe51-4f05-871f-69e92475fac1-kube-api-access-5smn9" (OuterVolumeSpecName: "kube-api-access-5smn9") pod "6faf211e-fe51-4f05-871f-69e92475fac1" (UID: "6faf211e-fe51-4f05-871f-69e92475fac1"). InnerVolumeSpecName "kube-api-access-5smn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:25.218024 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.217998    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:25.218024 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.218021    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf211e-fe51-4f05-871f-69e92475fac1-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:25.218193 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.218032    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:25.218193 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.218040    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:25.218193 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.218047    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf211e-fe51-4f05-871f-69e92475fac1-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:25.218193 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.218055    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smn9\" (UniqueName: \"kubernetes.io/projected/6faf211e-fe51-4f05-871f-69e92475fac1-kube-api-access-5smn9\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:25.253721 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.253692    2583 generic.go:356] "Generic (PLEG): container finished" podID="6faf211e-fe51-4f05-871f-69e92475fac1" containerID="c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342" exitCode=0
Sep 18 20:24:25.253875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.253742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2" event={"ID":"6faf211e-fe51-4f05-871f-69e92475fac1","Type":"ContainerDied","Data":"c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342"}
Sep 18 20:24:25.253875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.253751    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"
Sep 18 20:24:25.253875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.253797    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2" event={"ID":"6faf211e-fe51-4f05-871f-69e92475fac1","Type":"ContainerDied","Data":"0d5f58a3021a2db107068f03f5770f784e7c79d59ec8da5850d95e67ab7ac2a9"}
Sep 18 20:24:25.253875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.253820    2583 scope.go:117] "RemoveContainer" containerID="c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342"
Sep 18 20:24:25.262379 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.262336    2583 scope.go:117] "RemoveContainer" containerID="c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342"
Sep 18 20:24:25.262657 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:24:25.262639    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342\": container with ID starting with c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342 not found: ID does not exist" containerID="c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342"
Sep 18 20:24:25.262706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.262668    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342"} err="failed to get container status \"c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342\": rpc error: code = NotFound desc = could not find container \"c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342\": container with ID starting with c21a277f76fc9be20abccd5d7fccdfa75404a4d484b637550febaf6a3d35d342 not found: ID does not exist"
Sep 18 20:24:25.274130 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.274110    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"]
Sep 18 20:24:25.278031 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.278012    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-tmsh2"]
Sep 18 20:24:25.966058 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:25.966025    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6faf211e-fe51-4f05-871f-69e92475fac1" path="/var/lib/kubelet/pods/6faf211e-fe51-4f05-871f-69e92475fac1/volumes"
Sep 18 20:24:45.991053 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:45.990966    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"]
Sep 18 20:24:46.000099 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:24:46.000061    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:24:46.000099 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:24:46.000099 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:24:46.000099 ip-10-0-133-25 kubenswrapper[2583]:  > containerID="be9d52890ed5b7f453661a54e13259c11e02476c9027d7c78890b0c3b5ae481d" cmd=["/bin/sleep","15"]
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.000120    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" message="" err=<
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]:  >
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:24:46.000136    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" podUID="db18856a-9aca-4de6-bb5b-bddd57029bd2" containerName="main" containerID="cri-o://be9d52890ed5b7f453661a54e13259c11e02476c9027d7c78890b0c3b5ae481d"
Sep 18 20:24:46.000411 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.000178    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" podUID="db18856a-9aca-4de6-bb5b-bddd57029bd2" containerName="main" containerID="cri-o://be9d52890ed5b7f453661a54e13259c11e02476c9027d7c78890b0c3b5ae481d" gracePeriod=60
Sep 18 20:24:46.104700 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.104662    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" podUID="db18856a-9aca-4de6-bb5b-bddd57029bd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 18 20:24:46.353371 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.353227    2583 generic.go:356] "Generic (PLEG): container finished" podID="db18856a-9aca-4de6-bb5b-bddd57029bd2" containerID="be9d52890ed5b7f453661a54e13259c11e02476c9027d7c78890b0c3b5ae481d" exitCode=0
Sep 18 20:24:46.353371 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.353281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" event={"ID":"db18856a-9aca-4de6-bb5b-bddd57029bd2","Type":"ContainerDied","Data":"be9d52890ed5b7f453661a54e13259c11e02476c9027d7c78890b0c3b5ae481d"}
Sep 18 20:24:46.406459 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.406436    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:24:46.496213 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496185    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-model-cache\") pod \"db18856a-9aca-4de6-bb5b-bddd57029bd2\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") "
Sep 18 20:24:46.496213 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496220    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18856a-9aca-4de6-bb5b-bddd57029bd2-tls-certs\") pod \"db18856a-9aca-4de6-bb5b-bddd57029bd2\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") "
Sep 18 20:24:46.496487 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-home\") pod \"db18856a-9aca-4de6-bb5b-bddd57029bd2\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") "
Sep 18 20:24:46.496487 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-dshm\") pod \"db18856a-9aca-4de6-bb5b-bddd57029bd2\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") "
Sep 18 20:24:46.496487 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496420    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-tmp-dir\") pod \"db18856a-9aca-4de6-bb5b-bddd57029bd2\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") "
Sep 18 20:24:46.496487 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "db18856a-9aca-4de6-bb5b-bddd57029bd2" (UID: "db18856a-9aca-4de6-bb5b-bddd57029bd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:46.496698 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp5x4\" (UniqueName: \"kubernetes.io/projected/db18856a-9aca-4de6-bb5b-bddd57029bd2-kube-api-access-qp5x4\") pod \"db18856a-9aca-4de6-bb5b-bddd57029bd2\" (UID: \"db18856a-9aca-4de6-bb5b-bddd57029bd2\") "
Sep 18 20:24:46.496698 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496551    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-home" (OuterVolumeSpecName: "home") pod "db18856a-9aca-4de6-bb5b-bddd57029bd2" (UID: "db18856a-9aca-4de6-bb5b-bddd57029bd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:46.496698 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496639    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db18856a-9aca-4de6-bb5b-bddd57029bd2" (UID: "db18856a-9aca-4de6-bb5b-bddd57029bd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:46.496857 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496808    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:46.496857 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496827    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:46.496857 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.496841    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:46.498793 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.498765    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db18856a-9aca-4de6-bb5b-bddd57029bd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db18856a-9aca-4de6-bb5b-bddd57029bd2" (UID: "db18856a-9aca-4de6-bb5b-bddd57029bd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:46.498793 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.498783    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-dshm" (OuterVolumeSpecName: "dshm") pod "db18856a-9aca-4de6-bb5b-bddd57029bd2" (UID: "db18856a-9aca-4de6-bb5b-bddd57029bd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:46.498953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.498822    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db18856a-9aca-4de6-bb5b-bddd57029bd2-kube-api-access-qp5x4" (OuterVolumeSpecName: "kube-api-access-qp5x4") pod "db18856a-9aca-4de6-bb5b-bddd57029bd2" (UID: "db18856a-9aca-4de6-bb5b-bddd57029bd2"). InnerVolumeSpecName "kube-api-access-qp5x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:46.597939 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.597866    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18856a-9aca-4de6-bb5b-bddd57029bd2-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:46.597939 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.597893    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5x4\" (UniqueName: \"kubernetes.io/projected/db18856a-9aca-4de6-bb5b-bddd57029bd2-kube-api-access-qp5x4\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:46.597939 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:46.597916    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18856a-9aca-4de6-bb5b-bddd57029bd2-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:24:47.361113 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:47.361075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj" event={"ID":"db18856a-9aca-4de6-bb5b-bddd57029bd2","Type":"ContainerDied","Data":"6aacc08d9cc569c24f41ca9e9a8aba50cecd7741cecdee101c133b51e4d360f1"}
Sep 18 20:24:47.361709 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:47.361688    2583 scope.go:117] "RemoveContainer" containerID="be9d52890ed5b7f453661a54e13259c11e02476c9027d7c78890b0c3b5ae481d"
Sep 18 20:24:47.361996 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:47.361977    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"
Sep 18 20:24:47.391203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:47.391173    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"]
Sep 18 20:24:47.396694 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:47.396673    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-7m7mj"]
Sep 18 20:24:47.971845 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:47.971807    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db18856a-9aca-4de6-bb5b-bddd57029bd2" path="/var/lib/kubelet/pods/db18856a-9aca-4de6-bb5b-bddd57029bd2/volumes"
Sep 18 20:24:52.387760 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:52.387723    2583 generic.go:356] "Generic (PLEG): container finished" podID="8d059d02-fd40-4515-9356-bc207f394dd0" containerID="97b8998f3242f47efb3fbac95cede838b43ae3559ebc0a10a7a1d0a8623aa8dc" exitCode=0
Sep 18 20:24:52.388300 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:52.387801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" event={"ID":"8d059d02-fd40-4515-9356-bc207f394dd0","Type":"ContainerDied","Data":"97b8998f3242f47efb3fbac95cede838b43ae3559ebc0a10a7a1d0a8623aa8dc"}
Sep 18 20:24:52.388300 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:52.388292    2583 scope.go:117] "RemoveContainer" containerID="97b8998f3242f47efb3fbac95cede838b43ae3559ebc0a10a7a1d0a8623aa8dc"
Sep 18 20:24:52.389233 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:52.389201    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:24:52.389324 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:52.389271    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:24:52.389324 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:52.389302    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:24:53.393926 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:53.393887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" event={"ID":"8d059d02-fd40-4515-9356-bc207f394dd0","Type":"ContainerStarted","Data":"f8ed1855b497e037be9b922f21da7212d58ebd896afe1a753d56973ff6945391"}
Sep 18 20:24:54.363446 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:54.363406    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:54.363654 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:54.363532    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 18 20:24:55.192198 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:55.192167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:55.362084 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:55.362041    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 18 20:24:56.362401 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:56.362327    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 18 20:24:57.362161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:57.362111    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 18 20:24:58.362561 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:58.362513    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 18 20:24:59.366523 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:59.366495    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:24:59.367427 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:24:59.367405    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:25:16.562863 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.562833    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"]
Sep 18 20:25:16.563414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.563099    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf211e-fe51-4f05-871f-69e92475fac1" containerName="main"
Sep 18 20:25:16.563414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.563109    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf211e-fe51-4f05-871f-69e92475fac1" containerName="main"
Sep 18 20:25:16.563414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.563124    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db18856a-9aca-4de6-bb5b-bddd57029bd2" containerName="main"
Sep 18 20:25:16.563414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.563129    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="db18856a-9aca-4de6-bb5b-bddd57029bd2" containerName="main"
Sep 18 20:25:16.563414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.563177    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="db18856a-9aca-4de6-bb5b-bddd57029bd2" containerName="main"
Sep 18 20:25:16.563414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.563186    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6faf211e-fe51-4f05-871f-69e92475fac1" containerName="main"
Sep 18 20:25:16.565055 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.565038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.569497 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.569467    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 18 20:25:16.569497 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.569482    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 18 20:25:16.569925 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.569911    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-26l6j\""
Sep 18 20:25:16.570395 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.570379    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:25:16.570508 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.570492    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 18 20:25:16.571192 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.571172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 18 20:25:16.581739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.581716    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"]
Sep 18 20:25:16.652847 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.652819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.653008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.652866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.653008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.652888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.653008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.652935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.653008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.652970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.653008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.653003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.653207 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.653022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz84r\" (UniqueName: \"kubernetes.io/projected/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kube-api-access-zz84r\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.653207 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.653046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754063 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754063 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754251 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz84r\" (UniqueName: \"kubernetes.io/projected/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kube-api-access-zz84r\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754251 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754251 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754251 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754251 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754530 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754710 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754780 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754838 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.754838 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.755122 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.754897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.756566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.756547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.756697 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.756681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.763763 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.763736    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz84r\" (UniqueName: \"kubernetes.io/projected/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kube-api-access-zz84r\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.788766 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.788742    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"]
Sep 18 20:25:16.791124 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.791105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:16.794126 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.794107    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-2pjg5\""
Sep 18 20:25:16.804901 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.804871    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"]
Sep 18 20:25:16.855164 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.855095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993f02cb-7e7b-4196-a9cb-313eadd8abde-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:16.855164 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.855147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkvj\" (UniqueName: \"kubernetes.io/projected/993f02cb-7e7b-4196-a9cb-313eadd8abde-kube-api-access-trkvj\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:16.876114 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.876092    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:16.956078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.956044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993f02cb-7e7b-4196-a9cb-313eadd8abde-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:16.956224 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.956099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkvj\" (UniqueName: \"kubernetes.io/projected/993f02cb-7e7b-4196-a9cb-313eadd8abde-kube-api-access-trkvj\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:16.958813 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.958786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993f02cb-7e7b-4196-a9cb-313eadd8abde-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:16.967610 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:16.967590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkvj\" (UniqueName: \"kubernetes.io/projected/993f02cb-7e7b-4196-a9cb-313eadd8abde-kube-api-access-trkvj\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:17.101787 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.101756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:17.160927 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.160608    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"]
Sep 18 20:25:17.162960 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:25:17.162928    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6580bb_b23e_416b_a5c6_1f24c5b2dd6e.slice/crio-253c7189f179543ca709116b433979df64b99c7373678badf13ba278284af915 WatchSource:0}: Error finding container 253c7189f179543ca709116b433979df64b99c7373678badf13ba278284af915: Status 404 returned error can't find the container with id 253c7189f179543ca709116b433979df64b99c7373678badf13ba278284af915
Sep 18 20:25:17.383954 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.383928    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"]
Sep 18 20:25:17.386186 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:25:17.386159    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993f02cb_7e7b_4196_a9cb_313eadd8abde.slice/crio-00597e246e8763c010b4ed0c6be41047c84d17c4d0494d76d6df3ef61a09e7fd WatchSource:0}: Error finding container 00597e246e8763c010b4ed0c6be41047c84d17c4d0494d76d6df3ef61a09e7fd: Status 404 returned error can't find the container with id 00597e246e8763c010b4ed0c6be41047c84d17c4d0494d76d6df3ef61a09e7fd
Sep 18 20:25:17.397816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.397793    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"]
Sep 18 20:25:17.398011 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.397971    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy" containerID="cri-o://f8ed1855b497e037be9b922f21da7212d58ebd896afe1a753d56973ff6945391" gracePeriod=30
Sep 18 20:25:17.492245 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.492202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" event={"ID":"993f02cb-7e7b-4196-a9cb-313eadd8abde","Type":"ContainerStarted","Data":"6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84"}
Sep 18 20:25:17.492426 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.492278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" event={"ID":"993f02cb-7e7b-4196-a9cb-313eadd8abde","Type":"ContainerStarted","Data":"00597e246e8763c010b4ed0c6be41047c84d17c4d0494d76d6df3ef61a09e7fd"}
Sep 18 20:25:17.492697 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.492661    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:17.495019 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.494948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" event={"ID":"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e","Type":"ContainerStarted","Data":"c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2"}
Sep 18 20:25:17.495019 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.495003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" event={"ID":"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e","Type":"ContainerStarted","Data":"253c7189f179543ca709116b433979df64b99c7373678badf13ba278284af915"}
Sep 18 20:25:17.514843 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:17.514753    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" podStartSLOduration=1.5147341669999999 podStartE2EDuration="1.514734167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:17.51347141 +0000 UTC m=+2811.767678026" watchObservedRunningTime="2026-09-18 20:25:17.514734167 +0000 UTC m=+2811.768940771"
Sep 18 20:25:19.504174 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:19.504143    2583 generic.go:356] "Generic (PLEG): container finished" podID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerID="c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2" exitCode=0
Sep 18 20:25:19.504585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:19.504226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" event={"ID":"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e","Type":"ContainerDied","Data":"c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2"}
Sep 18 20:25:20.513284 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:20.513251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" event={"ID":"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e","Type":"ContainerStarted","Data":"2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53"}
Sep 18 20:25:20.537850 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:20.537797    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podStartSLOduration=2.4201239230000002 podStartE2EDuration="4.537782972s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.117659049s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:20.535893389 +0000 UTC m=+2814.790100007" watchObservedRunningTime="2026-09-18 20:25:20.537782972 +0000 UTC m=+2814.791989575"
Sep 18 20:25:20.876843 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:20.876752    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:22.524691 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.524659    2583 generic.go:356] "Generic (PLEG): container finished" podID="8d059d02-fd40-4515-9356-bc207f394dd0" containerID="f8ed1855b497e037be9b922f21da7212d58ebd896afe1a753d56973ff6945391" exitCode=0
Sep 18 20:25:22.525110 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.524723    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" event={"ID":"8d059d02-fd40-4515-9356-bc207f394dd0","Type":"ContainerDied","Data":"f8ed1855b497e037be9b922f21da7212d58ebd896afe1a753d56973ff6945391"}
Sep 18 20:25:22.525110 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.524784    2583 scope.go:117] "RemoveContainer" containerID="97b8998f3242f47efb3fbac95cede838b43ae3559ebc0a10a7a1d0a8623aa8dc"
Sep 18 20:25:22.644769 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.644744    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:25:22.705741 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.705709    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlkfb\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-kube-api-access-zlkfb\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.705909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.705766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-data\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.705909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.705792    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-istio-token\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.705909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.705821    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-credential-socket\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.705909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.705866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-socket\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.705909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.705892    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d059d02-fd40-4515-9356-bc207f394dd0-istio-podinfo\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.706154 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.705917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d059d02-fd40-4515-9356-bc207f394dd0-istiod-ca-cert\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.706154 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.706000    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-envoy\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.706154 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.706038    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-certs\") pod \"8d059d02-fd40-4515-9356-bc207f394dd0\" (UID: \"8d059d02-fd40-4515-9356-bc207f394dd0\") "
Sep 18 20:25:22.706433 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.706402    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:22.706857 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.706419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:22.706857 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.706431    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:22.707081 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.706843    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:22.707081 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.706943    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d059d02-fd40-4515-9356-bc207f394dd0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:22.709115 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.709086    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d059d02-fd40-4515-9356-bc207f394dd0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:25:22.709575 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.709544    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-kube-api-access-zlkfb" (OuterVolumeSpecName: "kube-api-access-zlkfb") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "kube-api-access-zlkfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:22.709773 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.709686    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:22.709773 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.709743    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d059d02-fd40-4515-9356-bc207f394dd0" (UID: "8d059d02-fd40-4515-9356-bc207f394dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:22.806665 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806630    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-data\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806665 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806658    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-istio-token\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806665 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806668    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-credential-socket\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806897 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806677    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-socket\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806897 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806685    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d059d02-fd40-4515-9356-bc207f394dd0-istio-podinfo\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806897 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806692    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d059d02-fd40-4515-9356-bc207f394dd0-istiod-ca-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806897 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806700    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-istio-envoy\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806897 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806707    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d059d02-fd40-4515-9356-bc207f394dd0-workload-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:22.806897 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:22.806714    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlkfb\" (UniqueName: \"kubernetes.io/projected/8d059d02-fd40-4515-9356-bc207f394dd0-kube-api-access-zlkfb\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:25:23.531878 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:23.531847    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"
Sep 18 20:25:23.531878 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:23.531865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd" event={"ID":"8d059d02-fd40-4515-9356-bc207f394dd0","Type":"ContainerDied","Data":"eb6b4e4cc44fff9d491e78cf1015d06082893c44a0da5d33facf261042bfc6e4"}
Sep 18 20:25:23.532428 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:23.531918    2583 scope.go:117] "RemoveContainer" containerID="f8ed1855b497e037be9b922f21da7212d58ebd896afe1a753d56973ff6945391"
Sep 18 20:25:23.553275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:23.553251    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"]
Sep 18 20:25:23.557936 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:23.557909    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-sw2xd"]
Sep 18 20:25:23.968861 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:23.968781    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" path="/var/lib/kubelet/pods/8d059d02-fd40-4515-9356-bc207f394dd0/volumes"
Sep 18 20:25:26.876316 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:26.876271    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:25:26.878043 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:26.878010    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:25:36.876704 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:36.876657    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:25:46.877478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:46.877435    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:25:48.501138 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:48.501107    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:25:56.876640 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:25:56.876590    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:26:06.876768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:26:06.876723    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:26:16.877303 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:26:16.877258    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:26:26.877250 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:26:26.877206    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:26:36.877078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:26:36.877033    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:26:46.876699 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:26:46.876651    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:26:56.876665 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:26:56.876620    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:27:06.877545 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:06.877501    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:27:16.886702 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:16.886672    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:27:16.894462 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:16.894430    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:27:21.713563 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:21.713521    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"]
Sep 18 20:27:21.717711 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:21.717683    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"]
Sep 18 20:27:28.501011 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:28.500972    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:27:36.736691 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:36.736639    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main" containerID="cri-o://6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84" gracePeriod=45
Sep 18 20:27:36.741675 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:36.741644    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" containerID="cri-o://2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53" gracePeriod=45
Sep 18 20:27:36.887638 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:36.887595    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 20:27:37.403059 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.403031    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:27:37.476507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476432    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-home\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.476507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476475    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tls-certs\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.476711 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tmp-dir\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.476711 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476570    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-model-cache\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.476711 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476597    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kserve-provision-location\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.476711 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476622    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-dshm\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.476711 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-cabundle-cert\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.476711 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz84r\" (UniqueName: \"kubernetes.io/projected/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kube-api-access-zz84r\") pod \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\" (UID: \"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e\") "
Sep 18 20:27:37.477015 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476847    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:37.477015 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.476959    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:37.477122 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.477074    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:37.477183 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.477131    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-home" (OuterVolumeSpecName: "home") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:37.479040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.479012    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:37.479158 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.479085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kube-api-access-zz84r" (OuterVolumeSpecName: "kube-api-access-zz84r") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "kube-api-access-zz84r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:27:37.479538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.479521    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-dshm" (OuterVolumeSpecName: "dshm") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:37.494679 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.494628    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:37.498065 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.498040    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" (UID: "5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:37.577982 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.577959    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:37.577982 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.577982    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:37.578125 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.577992    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:37.578125 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.578001    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz84r\" (UniqueName: \"kubernetes.io/projected/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-kube-api-access-zz84r\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:37.578125 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.578009    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:37.578125 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.578016    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:37.578125 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:37.578023    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:27:38.131380 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.131332    2583 generic.go:356] "Generic (PLEG): container finished" podID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerID="2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53" exitCode=0
Sep 18 20:27:38.131758 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.131405    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"
Sep 18 20:27:38.131758 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.131400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" event={"ID":"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e","Type":"ContainerDied","Data":"2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53"}
Sep 18 20:27:38.131758 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.131535    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f" event={"ID":"5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e","Type":"ContainerDied","Data":"253c7189f179543ca709116b433979df64b99c7373678badf13ba278284af915"}
Sep 18 20:27:38.131758 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.131564    2583 scope.go:117] "RemoveContainer" containerID="2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53"
Sep 18 20:27:38.139546 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.139512    2583 scope.go:117] "RemoveContainer" containerID="c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2"
Sep 18 20:27:38.147042 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.147024    2583 scope.go:117] "RemoveContainer" containerID="2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53"
Sep 18 20:27:38.147287 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:27:38.147269    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53\": container with ID starting with 2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53 not found: ID does not exist" containerID="2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53"
Sep 18 20:27:38.147336 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.147296    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53"} err="failed to get container status \"2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53\": rpc error: code = NotFound desc = could not find container \"2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53\": container with ID starting with 2997ac2a9e7559f282cdf6e58dac8b89dc39acea949d23532605a536b35a2c53 not found: ID does not exist"
Sep 18 20:27:38.147336 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.147313    2583 scope.go:117] "RemoveContainer" containerID="c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2"
Sep 18 20:27:38.147582 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:27:38.147561    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2\": container with ID starting with c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2 not found: ID does not exist" containerID="c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2"
Sep 18 20:27:38.147627 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.147588    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2"} err="failed to get container status \"c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2\": rpc error: code = NotFound desc = could not find container \"c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2\": container with ID starting with c551c1e33c49b37aef08ba92cbe1bcd818a1909209f6dab1903814cbe6d676f2 not found: ID does not exist"
Sep 18 20:27:38.164423 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.159920    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"]
Sep 18 20:27:38.170344 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.170314    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5f9b85b6b8vm89f"]
Sep 18 20:27:38.501221 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:38.501185    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:27:39.966302 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:39.966264    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" path="/var/lib/kubelet/pods/5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e/volumes"
Sep 18 20:27:48.500842 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:48.500800    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:27:48.501247 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:48.500898    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:27:58.500810 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:27:58.500768    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:28:06.975615 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:06.975587    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:28:07.016798 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.016770    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993f02cb-7e7b-4196-a9cb-313eadd8abde-tls-certs\") pod \"993f02cb-7e7b-4196-a9cb-313eadd8abde\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") "
Sep 18 20:28:07.016923 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.016866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trkvj\" (UniqueName: \"kubernetes.io/projected/993f02cb-7e7b-4196-a9cb-313eadd8abde-kube-api-access-trkvj\") pod \"993f02cb-7e7b-4196-a9cb-313eadd8abde\" (UID: \"993f02cb-7e7b-4196-a9cb-313eadd8abde\") "
Sep 18 20:28:07.018920 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.018889    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993f02cb-7e7b-4196-a9cb-313eadd8abde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "993f02cb-7e7b-4196-a9cb-313eadd8abde" (UID: "993f02cb-7e7b-4196-a9cb-313eadd8abde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:28:07.019020 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.018983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993f02cb-7e7b-4196-a9cb-313eadd8abde-kube-api-access-trkvj" (OuterVolumeSpecName: "kube-api-access-trkvj") pod "993f02cb-7e7b-4196-a9cb-313eadd8abde" (UID: "993f02cb-7e7b-4196-a9cb-313eadd8abde"). InnerVolumeSpecName "kube-api-access-trkvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:28:07.118080 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.118053    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkvj\" (UniqueName: \"kubernetes.io/projected/993f02cb-7e7b-4196-a9cb-313eadd8abde-kube-api-access-trkvj\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:28:07.118080 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.118076    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993f02cb-7e7b-4196-a9cb-313eadd8abde-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:28:07.252431 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.252394    2583 generic.go:356] "Generic (PLEG): container finished" podID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerID="6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84" exitCode=0
Sep 18 20:28:07.252566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.252441    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"
Sep 18 20:28:07.252566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.252464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" event={"ID":"993f02cb-7e7b-4196-a9cb-313eadd8abde","Type":"ContainerDied","Data":"6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84"}
Sep 18 20:28:07.252566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.252525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5" event={"ID":"993f02cb-7e7b-4196-a9cb-313eadd8abde","Type":"ContainerDied","Data":"00597e246e8763c010b4ed0c6be41047c84d17c4d0494d76d6df3ef61a09e7fd"}
Sep 18 20:28:07.252566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.252550    2583 scope.go:117] "RemoveContainer" containerID="6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84"
Sep 18 20:28:07.260780 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.260763    2583 scope.go:117] "RemoveContainer" containerID="6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84"
Sep 18 20:28:07.261031 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:28:07.261015    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84\": container with ID starting with 6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84 not found: ID does not exist" containerID="6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84"
Sep 18 20:28:07.261078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.261037    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84"} err="failed to get container status \"6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84\": rpc error: code = NotFound desc = could not find container \"6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84\": container with ID starting with 6ee404471fb3e91cb83b22a080cf62fae2b868b3f06d690452343de6832e3c84 not found: ID does not exist"
Sep 18 20:28:07.274422 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.274371    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"]
Sep 18 20:28:07.280078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.280058    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schemjqs5"]
Sep 18 20:28:07.965814 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:07.965779    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" path="/var/lib/kubelet/pods/993f02cb-7e7b-4196-a9cb-313eadd8abde/volumes"
Sep 18 20:28:18.332676 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332642    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"]
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332937    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332949    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332959    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332965    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332977    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="storage-initializer"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332982    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="storage-initializer"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332992    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.332997    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.333005    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.333010    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.333050    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="993f02cb-7e7b-4196-a9cb-313eadd8abde" containerName="main"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.333057    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy"
Sep 18 20:28:18.333142 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.333063    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6580bb-b23e-416b-a5c6-1f24c5b2dd6e" containerName="main"
Sep 18 20:28:18.333582 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.333151    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d059d02-fd40-4515-9356-bc207f394dd0" containerName="istio-proxy"
Sep 18 20:28:18.337487 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.337470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.340516 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.340493    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-gchg9\""
Sep 18 20:28:18.340718 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.340705    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 18 20:28:18.342398 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.342379    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 18 20:28:18.342608 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.342585    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 18 20:28:18.355045 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.355024    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"]
Sep 18 20:28:18.399087 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.399063    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.399210 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.399105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m26d\" (UniqueName: \"kubernetes.io/projected/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kube-api-access-6m26d\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.399210 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.399177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.399296 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.399226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.399296 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.399246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.399296 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.399278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.399454 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.399303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.499798 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.499760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.499798 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.499804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.499991 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.499822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.499991 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.499856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.499991 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.499888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.499991 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.499918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.499991 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.499959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m26d\" (UniqueName: \"kubernetes.io/projected/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kube-api-access-6m26d\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.500247 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.500182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.500247 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.500224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.500458 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.500436    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.500667 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.500555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.502256 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.502238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.502463 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.502446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.508338 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.508315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m26d\" (UniqueName: \"kubernetes.io/projected/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kube-api-access-6m26d\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.646553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.646475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:18.922707 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:18.922620    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"]
Sep 18 20:28:19.305243 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:19.305209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerStarted","Data":"7a08c8b1d47f8cff54c7c9e16c4d7daf1010158c2a0de53ef37cf20cbaea71e8"}
Sep 18 20:28:26.339630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:26.339592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerStarted","Data":"f17395c1ca1ae460a8b4c2ab7befa47939ed0c90e79c28a40ce396caca62764d"}
Sep 18 20:28:27.345960 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:27.345926    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerID="f17395c1ca1ae460a8b4c2ab7befa47939ed0c90e79c28a40ce396caca62764d" exitCode=0
Sep 18 20:28:27.346381 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:27.345972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerDied","Data":"f17395c1ca1ae460a8b4c2ab7befa47939ed0c90e79c28a40ce396caca62764d"}
Sep 18 20:28:28.079518 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.079491    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:28:28.083024 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.083002    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:28:28.352732 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.352697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerStarted","Data":"72ad835d5580aa1dc4f120fdb9f0014ad495c34d2ab9c7ba80c83ae3bd9de0db"}
Sep 18 20:28:28.353219 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.352751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerStarted","Data":"8c0ba0ebe607c407fe78c12e41c6b7e942006d9897c500e870c97083578b7919"}
Sep 18 20:28:28.379520 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.379472    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podStartSLOduration=2.226009363 podStartE2EDuration="10.379455688s" totalImagesPullingTime="7.066158466s" totalInitContainerRuntime="1.087287859s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:28.377616664 +0000 UTC m=+3002.631823278" watchObservedRunningTime="2026-09-18 20:28:28.379455688 +0000 UTC m=+3002.633662291"
Sep 18 20:28:28.647402 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.647376    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:28.647604 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.647475    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:28:28.648750 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:28.648716    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:28:38.647333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:38.647288    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:28:48.647109 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:48.647071    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:28:58.647194 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:28:58.647154    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:29:08.647120 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:29:08.647035    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:29:18.647063 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:29:18.647020    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:29:28.646896 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:29:28.646853    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:29:38.647281 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:29:38.647236    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:29:48.646836 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:29:48.646792    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:29:58.646840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:29:58.646793    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:30:08.646952 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:08.646903    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:30:18.658134 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:18.658097    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:30:18.665928 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:18.665899    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:30:23.970588 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:23.970556    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"]
Sep 18 20:30:23.971183 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:23.970834    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="modelcar" containerID="cri-o://72ad835d5580aa1dc4f120fdb9f0014ad495c34d2ab9c7ba80c83ae3bd9de0db" gracePeriod=60
Sep 18 20:30:24.877154 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:24.877124    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv_6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89/modelcar/0.log"
Sep 18 20:30:24.877857 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:24.877828    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerID="72ad835d5580aa1dc4f120fdb9f0014ad495c34d2ab9c7ba80c83ae3bd9de0db" exitCode=143
Sep 18 20:30:24.877981 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:24.877878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerDied","Data":"72ad835d5580aa1dc4f120fdb9f0014ad495c34d2ab9c7ba80c83ae3bd9de0db"}
Sep 18 20:30:38.993713 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:38.993629    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" containerID="cri-o://8c0ba0ebe607c407fe78c12e41c6b7e942006d9897c500e870c97083578b7919" gracePeriod=45
Sep 18 20:30:39.659053 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:39.659012    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 20:30:39.936809 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:39.936779    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv_6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89/modelcar/0.log"
Sep 18 20:30:39.937489 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:39.937462    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerID="8c0ba0ebe607c407fe78c12e41c6b7e942006d9897c500e870c97083578b7919" exitCode=0
Sep 18 20:30:39.937616 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:39.937536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerDied","Data":"8c0ba0ebe607c407fe78c12e41c6b7e942006d9897c500e870c97083578b7919"}
Sep 18 20:30:40.243833 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.243810    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv_6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89/modelcar/0.log"
Sep 18 20:30:40.246707 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.246688    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:30:40.320601 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320575    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-dshm\") pod \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") "
Sep 18 20:30:40.320765 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320616    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-home\") pod \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") "
Sep 18 20:30:40.320765 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320631    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-model-cache\") pod \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") "
Sep 18 20:30:40.320765 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320684    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tmp-dir\") pod \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") "
Sep 18 20:30:40.320765 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m26d\" (UniqueName: \"kubernetes.io/projected/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kube-api-access-6m26d\") pod \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") "
Sep 18 20:30:40.320765 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320735    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tls-certs\") pod \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") "
Sep 18 20:30:40.321044 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kserve-provision-location\") pod \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\" (UID: \"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89\") "
Sep 18 20:30:40.321044 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.320903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" (UID: "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:40.321198 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.321174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" (UID: "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:40.321305 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.321285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-home" (OuterVolumeSpecName: "home") pod "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" (UID: "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:40.323062 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.323031    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kube-api-access-6m26d" (OuterVolumeSpecName: "kube-api-access-6m26d") pod "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" (UID: "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89"). InnerVolumeSpecName "kube-api-access-6m26d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:40.323447 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.323419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-dshm" (OuterVolumeSpecName: "dshm") pod "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" (UID: "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:40.323553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.323462    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" (UID: "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:40.331926 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.331899    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" (UID: "6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:40.421643 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.421615    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:30:40.421643 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.421640    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m26d\" (UniqueName: \"kubernetes.io/projected/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kube-api-access-6m26d\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:30:40.421800 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.421650    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:30:40.421800 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.421660    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:30:40.421800 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.421668    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:30:40.421800 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.421675    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:30:40.421800 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.421682    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:30:40.942730 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.942702    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv_6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89/modelcar/0.log"
Sep 18 20:30:40.943498 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.943469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv" event={"ID":"6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89","Type":"ContainerDied","Data":"7a08c8b1d47f8cff54c7c9e16c4d7daf1010158c2a0de53ef37cf20cbaea71e8"}
Sep 18 20:30:40.943569 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.943543    2583 scope.go:117] "RemoveContainer" containerID="72ad835d5580aa1dc4f120fdb9f0014ad495c34d2ab9c7ba80c83ae3bd9de0db"
Sep 18 20:30:40.943608 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.943482    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"
Sep 18 20:30:40.952106 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.952088    2583 scope.go:117] "RemoveContainer" containerID="8c0ba0ebe607c407fe78c12e41c6b7e942006d9897c500e870c97083578b7919"
Sep 18 20:30:40.959649 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.959632    2583 scope.go:117] "RemoveContainer" containerID="f17395c1ca1ae460a8b4c2ab7befa47939ed0c90e79c28a40ce396caca62764d"
Sep 18 20:30:40.963336 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.963311    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"]
Sep 18 20:30:40.967899 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:40.967877    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8ffhrcv"]
Sep 18 20:30:41.964990 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:30:41.964958    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" path="/var/lib/kubelet/pods/6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89/volumes"
Sep 18 20:31:27.721509 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721473    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"]
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721768    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="modelcar-init"
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721779    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="modelcar-init"
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721786    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main"
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721791    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main"
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721812    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="modelcar"
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721818    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="modelcar"
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721857    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="main"
Sep 18 20:31:27.721983 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.721864    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce0d0fc-13cc-4570-aa8c-d84ac2d44c89" containerName="modelcar"
Sep 18 20:31:27.724567 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.724548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:27.728176 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.728148    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 20:31:27.728176 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.728171    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 20:31:27.728384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.728150    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-hc59z\""
Sep 18 20:31:27.728384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.728156    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 20:31:27.728384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.728162    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-rlfvg\""
Sep 18 20:31:27.733462 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.733440    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"]
Sep 18 20:31:27.869937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.869879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8951194b-269a-496a-a12d-f39f8c093744-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:27.869937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.869939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glpvx\" (UniqueName: \"kubernetes.io/projected/8951194b-269a-496a-a12d-f39f8c093744-kube-api-access-glpvx\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:27.970613 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.970581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8951194b-269a-496a-a12d-f39f8c093744-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:27.970613 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.970613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glpvx\" (UniqueName: \"kubernetes.io/projected/8951194b-269a-496a-a12d-f39f8c093744-kube-api-access-glpvx\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:27.973257 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.973190    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 20:31:27.978865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.978847    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 20:31:27.983988 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.983965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8951194b-269a-496a-a12d-f39f8c093744-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:27.989235 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.989205    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 20:31:27.999455 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:27.999437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glpvx\" (UniqueName: \"kubernetes.io/projected/8951194b-269a-496a-a12d-f39f8c093744-kube-api-access-glpvx\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:28.037914 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:28.037896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-rlfvg\""
Sep 18 20:31:28.048688 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:28.048660    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-hc59z\""
Sep 18 20:31:28.057654 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:28.057637    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:28.350573 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:28.350546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"]
Sep 18 20:31:28.353057 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:31:28.353030    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8951194b_269a_496a_a12d_f39f8c093744.slice/crio-b40b8a5a4a7df77bc5db404a703961de9205a3f1c1e586fab70f3e2023bc0c65 WatchSource:0}: Error finding container b40b8a5a4a7df77bc5db404a703961de9205a3f1c1e586fab70f3e2023bc0c65: Status 404 returned error can't find the container with id b40b8a5a4a7df77bc5db404a703961de9205a3f1c1e586fab70f3e2023bc0c65
Sep 18 20:31:29.131772 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:29.131737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" event={"ID":"8951194b-269a-496a-a12d-f39f8c093744","Type":"ContainerStarted","Data":"0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3"}
Sep 18 20:31:29.132160 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:29.131785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" event={"ID":"8951194b-269a-496a-a12d-f39f8c093744","Type":"ContainerStarted","Data":"b40b8a5a4a7df77bc5db404a703961de9205a3f1c1e586fab70f3e2023bc0c65"}
Sep 18 20:31:29.132160 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:29.131866    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:31:29.148423 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:31:29.148380    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" podStartSLOduration=2.148350196 podStartE2EDuration="2.148350196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:29.148254492 +0000 UTC m=+3183.402461096" watchObservedRunningTime="2026-09-18 20:31:29.148350196 +0000 UTC m=+3183.402556801"
Sep 18 20:32:00.137228 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:00.137197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:32:05.893173 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.893135    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"]
Sep 18 20:32:05.896530 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.896512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:05.899332 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.899307    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 18 20:32:05.899482 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.899313    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 18 20:32:05.899482 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.899401    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 18 20:32:05.899482 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.899418    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 18 20:32:05.899656 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.899565    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-d7xmz\""
Sep 18 20:32:05.910160 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.910138    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"]
Sep 18 20:32:05.943106 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.943078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:05.943258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.943130    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-dshm\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:05.943258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.943146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jwf\" (UniqueName: \"kubernetes.io/projected/3bc82f89-f764-4d3a-935d-ccb02c793906-kube-api-access-29jwf\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:05.943258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.943183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:05.943258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.943222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-home\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:05.943258 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.943248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc82f89-f764-4d3a-935d-ccb02c793906-tls-certs\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:05.943504 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:05.943264    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-model-cache\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.044581 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.044547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.044790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.044597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-home\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.044790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.044619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc82f89-f764-4d3a-935d-ccb02c793906-tls-certs\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.044790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.044637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-model-cache\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.044790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.044762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.045013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.044822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-dshm\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.045013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.044849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jwf\" (UniqueName: \"kubernetes.io/projected/3bc82f89-f764-4d3a-935d-ccb02c793906-kube-api-access-29jwf\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.045013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.045003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-model-cache\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.045164 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.045081    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.045164 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.045101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-home\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.045232 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.045177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.047228 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.047203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-dshm\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.047461 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.047444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc82f89-f764-4d3a-935d-ccb02c793906-tls-certs\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.057083 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.057059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jwf\" (UniqueName: \"kubernetes.io/projected/3bc82f89-f764-4d3a-935d-ccb02c793906-kube-api-access-29jwf\") pod \"e2e-del-blocked-svc-kserve-69458ff968-slmvn\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.207949 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.207863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:06.485315 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:06.485226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"]
Sep 18 20:32:06.488761 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:32:06.488725    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc82f89_f764_4d3a_935d_ccb02c793906.slice/crio-f48c35873925b808616bf392faece2d7a80dda39dee37da662cb026b65209f0f WatchSource:0}: Error finding container f48c35873925b808616bf392faece2d7a80dda39dee37da662cb026b65209f0f: Status 404 returned error can't find the container with id f48c35873925b808616bf392faece2d7a80dda39dee37da662cb026b65209f0f
Sep 18 20:32:07.288235 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:07.288202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn" event={"ID":"3bc82f89-f764-4d3a-935d-ccb02c793906","Type":"ContainerStarted","Data":"54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d"}
Sep 18 20:32:07.288638 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:07.288253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn" event={"ID":"3bc82f89-f764-4d3a-935d-ccb02c793906","Type":"ContainerStarted","Data":"f48c35873925b808616bf392faece2d7a80dda39dee37da662cb026b65209f0f"}
Sep 18 20:32:09.011569 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:09.011537    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"]
Sep 18 20:32:09.297793 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:09.297701    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn" podUID="3bc82f89-f764-4d3a-935d-ccb02c793906" containerName="storage-initializer" containerID="cri-o://54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d" gracePeriod=60
Sep 18 20:32:10.744346 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.744321    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:10.785562 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-dshm\") pod \"3bc82f89-f764-4d3a-935d-ccb02c793906\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") "
Sep 18 20:32:10.785680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785576    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc82f89-f764-4d3a-935d-ccb02c793906-tls-certs\") pod \"3bc82f89-f764-4d3a-935d-ccb02c793906\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") "
Sep 18 20:32:10.785680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785623    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-kserve-provision-location\") pod \"3bc82f89-f764-4d3a-935d-ccb02c793906\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") "
Sep 18 20:32:10.785680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785639    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-home\") pod \"3bc82f89-f764-4d3a-935d-ccb02c793906\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") "
Sep 18 20:32:10.785680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785656    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-tmp-dir\") pod \"3bc82f89-f764-4d3a-935d-ccb02c793906\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") "
Sep 18 20:32:10.785873 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785691    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29jwf\" (UniqueName: \"kubernetes.io/projected/3bc82f89-f764-4d3a-935d-ccb02c793906-kube-api-access-29jwf\") pod \"3bc82f89-f764-4d3a-935d-ccb02c793906\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") "
Sep 18 20:32:10.785873 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-model-cache\") pod \"3bc82f89-f764-4d3a-935d-ccb02c793906\" (UID: \"3bc82f89-f764-4d3a-935d-ccb02c793906\") "
Sep 18 20:32:10.786013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.785983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3bc82f89-f764-4d3a-935d-ccb02c793906" (UID: "3bc82f89-f764-4d3a-935d-ccb02c793906"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:10.786079 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.786009    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-home" (OuterVolumeSpecName: "home") pod "3bc82f89-f764-4d3a-935d-ccb02c793906" (UID: "3bc82f89-f764-4d3a-935d-ccb02c793906"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:10.786123 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.786076    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-model-cache" (OuterVolumeSpecName: "model-cache") pod "3bc82f89-f764-4d3a-935d-ccb02c793906" (UID: "3bc82f89-f764-4d3a-935d-ccb02c793906"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:10.787880 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.787860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc82f89-f764-4d3a-935d-ccb02c793906-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bc82f89-f764-4d3a-935d-ccb02c793906" (UID: "3bc82f89-f764-4d3a-935d-ccb02c793906"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:10.788084 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.788066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc82f89-f764-4d3a-935d-ccb02c793906-kube-api-access-29jwf" (OuterVolumeSpecName: "kube-api-access-29jwf") pod "3bc82f89-f764-4d3a-935d-ccb02c793906" (UID: "3bc82f89-f764-4d3a-935d-ccb02c793906"). InnerVolumeSpecName "kube-api-access-29jwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:10.788141 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.788103    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-dshm" (OuterVolumeSpecName: "dshm") pod "3bc82f89-f764-4d3a-935d-ccb02c793906" (UID: "3bc82f89-f764-4d3a-935d-ccb02c793906"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:10.857453 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.857331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bc82f89-f764-4d3a-935d-ccb02c793906" (UID: "3bc82f89-f764-4d3a-935d-ccb02c793906"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:10.887132 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.887096    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:10.887132 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.887124    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:10.887329 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.887155    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:10.887329 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.887164    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jwf\" (UniqueName: \"kubernetes.io/projected/3bc82f89-f764-4d3a-935d-ccb02c793906-kube-api-access-29jwf\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:10.887329 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.887173    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:10.887329 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.887181    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc82f89-f764-4d3a-935d-ccb02c793906-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:10.887329 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:10.887188    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc82f89-f764-4d3a-935d-ccb02c793906-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:11.308141 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.308103    2583 generic.go:356] "Generic (PLEG): container finished" podID="3bc82f89-f764-4d3a-935d-ccb02c793906" containerID="54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d" exitCode=0
Sep 18 20:32:11.308344 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.308150    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"
Sep 18 20:32:11.308344 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.308171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn" event={"ID":"3bc82f89-f764-4d3a-935d-ccb02c793906","Type":"ContainerDied","Data":"54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d"}
Sep 18 20:32:11.308344 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.308224    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn" event={"ID":"3bc82f89-f764-4d3a-935d-ccb02c793906","Type":"ContainerDied","Data":"f48c35873925b808616bf392faece2d7a80dda39dee37da662cb026b65209f0f"}
Sep 18 20:32:11.308344 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.308252    2583 scope.go:117] "RemoveContainer" containerID="54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d"
Sep 18 20:32:11.317036 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.317018    2583 scope.go:117] "RemoveContainer" containerID="54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d"
Sep 18 20:32:11.317295 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:32:11.317275    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d\": container with ID starting with 54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d not found: ID does not exist" containerID="54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d"
Sep 18 20:32:11.317342 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.317306    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d"} err="failed to get container status \"54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d\": rpc error: code = NotFound desc = could not find container \"54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d\": container with ID starting with 54baace5838ab4efa6dec726213816ab9d58dff8a3ed8a05fe79be9b3eb9fe5d not found: ID does not exist"
Sep 18 20:32:11.347010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.346980    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"]
Sep 18 20:32:11.354478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.354455    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69458ff968-slmvn"]
Sep 18 20:32:11.966820 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:11.966782    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc82f89-f764-4d3a-935d-ccb02c793906" path="/var/lib/kubelet/pods/3bc82f89-f764-4d3a-935d-ccb02c793906/volumes"
Sep 18 20:32:47.400003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.399968    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"]
Sep 18 20:32:47.400611 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.400239    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc82f89-f764-4d3a-935d-ccb02c793906" containerName="storage-initializer"
Sep 18 20:32:47.400611 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.400250    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc82f89-f764-4d3a-935d-ccb02c793906" containerName="storage-initializer"
Sep 18 20:32:47.400611 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.400303    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc82f89-f764-4d3a-935d-ccb02c793906" containerName="storage-initializer"
Sep 18 20:32:47.404467 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.404442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:47.407378 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.407327    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 18 20:32:47.407509 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.407383    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 18 20:32:47.407509 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.407415    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-pwd5r\""
Sep 18 20:32:47.407509 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.407415    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 18 20:32:47.412451 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.412430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"]
Sep 18 20:32:47.554675 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.554642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:47.554859 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.554695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42vj\" (UniqueName: \"kubernetes.io/projected/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kube-api-access-t42vj\") pod \"e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:47.656061 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.655966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42vj\" (UniqueName: \"kubernetes.io/projected/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kube-api-access-t42vj\") pod \"e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:47.656061 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.656024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:47.656451 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.656432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:47.665294 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.665264    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42vj\" (UniqueName: \"kubernetes.io/projected/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kube-api-access-t42vj\") pod \"e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:47.715189 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:47.715166    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:48.000589 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:48.000564    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"]
Sep 18 20:32:48.001602 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:32:48.001576    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60527ec5_80f0_434c_a593_8a2d70c1dfd8.slice/crio-2fbd2a08f61cca0c1d1a07286a2608f0fb84c3e751ba7bcf4524e17607cdd66e WatchSource:0}: Error finding container 2fbd2a08f61cca0c1d1a07286a2608f0fb84c3e751ba7bcf4524e17607cdd66e: Status 404 returned error can't find the container with id 2fbd2a08f61cca0c1d1a07286a2608f0fb84c3e751ba7bcf4524e17607cdd66e
Sep 18 20:32:48.461217 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:48.461180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f" event={"ID":"60527ec5-80f0-434c-a593-8a2d70c1dfd8","Type":"ContainerStarted","Data":"d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43"}
Sep 18 20:32:48.461657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:48.461243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f" event={"ID":"60527ec5-80f0-434c-a593-8a2d70c1dfd8","Type":"ContainerStarted","Data":"2fbd2a08f61cca0c1d1a07286a2608f0fb84c3e751ba7bcf4524e17607cdd66e"}
Sep 18 20:32:50.252170 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:50.252136    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"]
Sep 18 20:32:50.470170 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:50.470101    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f" podUID="60527ec5-80f0-434c-a593-8a2d70c1dfd8" containerName="storage-initializer" containerID="cri-o://d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43" gracePeriod=30
Sep 18 20:32:52.816369 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:52.816331    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:52.896923 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:52.896895    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t42vj\" (UniqueName: \"kubernetes.io/projected/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kube-api-access-t42vj\") pod \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") "
Sep 18 20:32:52.897063 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:52.896954    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kserve-provision-location\") pod \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\" (UID: \"60527ec5-80f0-434c-a593-8a2d70c1dfd8\") "
Sep 18 20:32:52.899180 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:52.899156    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kube-api-access-t42vj" (OuterVolumeSpecName: "kube-api-access-t42vj") pod "60527ec5-80f0-434c-a593-8a2d70c1dfd8" (UID: "60527ec5-80f0-434c-a593-8a2d70c1dfd8"). InnerVolumeSpecName "kube-api-access-t42vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:52.966814 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:52.966744    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60527ec5-80f0-434c-a593-8a2d70c1dfd8" (UID: "60527ec5-80f0-434c-a593-8a2d70c1dfd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:52.997714 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:52.997689    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:52.997714 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:52.997713    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t42vj\" (UniqueName: \"kubernetes.io/projected/60527ec5-80f0-434c-a593-8a2d70c1dfd8-kube-api-access-t42vj\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:32:53.483110 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.483075    2583 generic.go:356] "Generic (PLEG): container finished" podID="60527ec5-80f0-434c-a593-8a2d70c1dfd8" containerID="d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43" exitCode=0
Sep 18 20:32:53.483279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.483125    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"
Sep 18 20:32:53.483279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.483139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f" event={"ID":"60527ec5-80f0-434c-a593-8a2d70c1dfd8","Type":"ContainerDied","Data":"d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43"}
Sep 18 20:32:53.483279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.483191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f" event={"ID":"60527ec5-80f0-434c-a593-8a2d70c1dfd8","Type":"ContainerDied","Data":"2fbd2a08f61cca0c1d1a07286a2608f0fb84c3e751ba7bcf4524e17607cdd66e"}
Sep 18 20:32:53.483279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.483220    2583 scope.go:117] "RemoveContainer" containerID="d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43"
Sep 18 20:32:53.492284 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.492265    2583 scope.go:117] "RemoveContainer" containerID="d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43"
Sep 18 20:32:53.492583 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:32:53.492560    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43\": container with ID starting with d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43 not found: ID does not exist" containerID="d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43"
Sep 18 20:32:53.492637 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.492592    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43"} err="failed to get container status \"d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43\": rpc error: code = NotFound desc = could not find container \"d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43\": container with ID starting with d4bae843209b5392a6444429168ea8ee9b420ff29bc798636ba5d603b66e7f43 not found: ID does not exist"
Sep 18 20:32:53.517349 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.517325    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"]
Sep 18 20:32:53.526071 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.526037    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-586b9bd5f7-fdz5f"]
Sep 18 20:32:53.966365 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:32:53.966320    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60527ec5-80f0-434c-a593-8a2d70c1dfd8" path="/var/lib/kubelet/pods/60527ec5-80f0-434c-a593-8a2d70c1dfd8/volumes"
Sep 18 20:33:08.633905 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.633874    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"]
Sep 18 20:33:08.634280 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.634147    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60527ec5-80f0-434c-a593-8a2d70c1dfd8" containerName="storage-initializer"
Sep 18 20:33:08.634280 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.634157    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="60527ec5-80f0-434c-a593-8a2d70c1dfd8" containerName="storage-initializer"
Sep 18 20:33:08.634280 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.634201    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="60527ec5-80f0-434c-a593-8a2d70c1dfd8" containerName="storage-initializer"
Sep 18 20:33:08.636683 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.636662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.640527 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.640506    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:33:08.640749 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.640731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 20:33:08.640979 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.640963    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-g4sg4\""
Sep 18 20:33:08.641077 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.641061    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 20:33:08.641743 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.641729    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 20:33:08.641930 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.641912    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 20:33:08.658740 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.658716    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"]
Sep 18 20:33:08.714260 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a053100-0efe-46e8-bc49-cc16e936376c-cabundle-cert\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.714374 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdb6\" (UniqueName: \"kubernetes.io/projected/7a053100-0efe-46e8-bc49-cc16e936376c-kube-api-access-bzdb6\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.714374 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-home\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.714374 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-model-cache\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.714515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-dshm\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.714515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-tmp-dir\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.714515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a053100-0efe-46e8-bc49-cc16e936376c-tls-certs\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.714515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.714506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-kserve-provision-location\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.814789 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.814764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-home\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.814903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.814798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-model-cache\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.814903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.814819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-dshm\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.814903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.814843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-tmp-dir\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.814903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.814887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a053100-0efe-46e8-bc49-cc16e936376c-tls-certs\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.814920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-kserve-provision-location\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.814968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a053100-0efe-46e8-bc49-cc16e936376c-cabundle-cert\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815214 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.815098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdb6\" (UniqueName: \"kubernetes.io/projected/7a053100-0efe-46e8-bc49-cc16e936376c-kube-api-access-bzdb6\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815214 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.815187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-home\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815327 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.815275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-model-cache\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.815418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-kserve-provision-location\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815532 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.815511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-tmp-dir\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.815898 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.815877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a053100-0efe-46e8-bc49-cc16e936376c-cabundle-cert\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.817345 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.817319    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-dshm\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.817542 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.817524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a053100-0efe-46e8-bc49-cc16e936376c-tls-certs\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.823659 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.823634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdb6\" (UniqueName: \"kubernetes.io/projected/7a053100-0efe-46e8-bc49-cc16e936376c-kube-api-access-bzdb6\") pod \"stop-feature-test-kserve-768994c8cb-dlrxt\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:08.946814 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:08.946730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:09.230250 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:09.230220    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"]
Sep 18 20:33:09.232315 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:33:09.232287    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a053100_0efe_46e8_bc49_cc16e936376c.slice/crio-302d2587b2828f407c5c2d9faa9ddfe5dc274f24c8a6d19fe455164fb31eee8d WatchSource:0}: Error finding container 302d2587b2828f407c5c2d9faa9ddfe5dc274f24c8a6d19fe455164fb31eee8d: Status 404 returned error can't find the container with id 302d2587b2828f407c5c2d9faa9ddfe5dc274f24c8a6d19fe455164fb31eee8d
Sep 18 20:33:09.550845 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:09.550801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" event={"ID":"7a053100-0efe-46e8-bc49-cc16e936376c","Type":"ContainerStarted","Data":"a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d"}
Sep 18 20:33:09.551039 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:09.550876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" event={"ID":"7a053100-0efe-46e8-bc49-cc16e936376c","Type":"ContainerStarted","Data":"302d2587b2828f407c5c2d9faa9ddfe5dc274f24c8a6d19fe455164fb31eee8d"}
Sep 18 20:33:11.561982 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:11.561945    2583 generic.go:356] "Generic (PLEG): container finished" podID="7a053100-0efe-46e8-bc49-cc16e936376c" containerID="a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d" exitCode=0
Sep 18 20:33:11.562399 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:11.562021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" event={"ID":"7a053100-0efe-46e8-bc49-cc16e936376c","Type":"ContainerDied","Data":"a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d"}
Sep 18 20:33:12.569279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:12.569238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" event={"ID":"7a053100-0efe-46e8-bc49-cc16e936376c","Type":"ContainerStarted","Data":"fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604"}
Sep 18 20:33:12.592712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:12.592660    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podStartSLOduration=2.494561357 podStartE2EDuration="4.592645255s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.098083898s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:12.590201913 +0000 UTC m=+3286.844408512" watchObservedRunningTime="2026-09-18 20:33:12.592645255 +0000 UTC m=+3286.846851859"
Sep 18 20:33:12.947060 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:12.946961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:18.947067 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:18.947022    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:33:18.948585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:18.948557    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:33:28.105883 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:28.105849    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:33:28.109625 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:28.109605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:33:28.947623 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:28.947578    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:33:32.244606 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:32.244572    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"]
Sep 18 20:33:38.947506 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:38.947413    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:33:40.137854 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:40.137812    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:33:47.268013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:47.267956    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main" containerID="cri-o://0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3" gracePeriod=45
Sep 18 20:33:48.947497 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:48.947455    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:33:50.137462 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:50.137423    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:33:58.947183 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:33:58.947130    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:34:00.137992 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:00.137956    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:34:00.138413 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:00.138050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:34:08.947466 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:08.947423    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:34:10.137734 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:10.137697    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:34:17.513568 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.513538    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:34:17.577373 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.577269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glpvx\" (UniqueName: \"kubernetes.io/projected/8951194b-269a-496a-a12d-f39f8c093744-kube-api-access-glpvx\") pod \"8951194b-269a-496a-a12d-f39f8c093744\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") "
Sep 18 20:34:17.577373 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.577341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8951194b-269a-496a-a12d-f39f8c093744-tls-certs\") pod \"8951194b-269a-496a-a12d-f39f8c093744\" (UID: \"8951194b-269a-496a-a12d-f39f8c093744\") "
Sep 18 20:34:17.579729 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.579695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8951194b-269a-496a-a12d-f39f8c093744-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8951194b-269a-496a-a12d-f39f8c093744" (UID: "8951194b-269a-496a-a12d-f39f8c093744"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:34:17.579729 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.579708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8951194b-269a-496a-a12d-f39f8c093744-kube-api-access-glpvx" (OuterVolumeSpecName: "kube-api-access-glpvx") pod "8951194b-269a-496a-a12d-f39f8c093744" (UID: "8951194b-269a-496a-a12d-f39f8c093744"). InnerVolumeSpecName "kube-api-access-glpvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:34:17.678727 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.678698    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glpvx\" (UniqueName: \"kubernetes.io/projected/8951194b-269a-496a-a12d-f39f8c093744-kube-api-access-glpvx\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:34:17.678727 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.678722    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8951194b-269a-496a-a12d-f39f8c093744-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:34:17.853414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.853324    2583 generic.go:356] "Generic (PLEG): container finished" podID="8951194b-269a-496a-a12d-f39f8c093744" containerID="0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3" exitCode=0
Sep 18 20:34:17.853414 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.853395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" event={"ID":"8951194b-269a-496a-a12d-f39f8c093744","Type":"ContainerDied","Data":"0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3"}
Sep 18 20:34:17.853584 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.853413    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"
Sep 18 20:34:17.853584 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.853443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7" event={"ID":"8951194b-269a-496a-a12d-f39f8c093744","Type":"ContainerDied","Data":"b40b8a5a4a7df77bc5db404a703961de9205a3f1c1e586fab70f3e2023bc0c65"}
Sep 18 20:34:17.853584 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.853469    2583 scope.go:117] "RemoveContainer" containerID="0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3"
Sep 18 20:34:17.861774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.861752    2583 scope.go:117] "RemoveContainer" containerID="0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3"
Sep 18 20:34:17.862055 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:34:17.862036    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3\": container with ID starting with 0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3 not found: ID does not exist" containerID="0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3"
Sep 18 20:34:17.862117 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.862067    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3"} err="failed to get container status \"0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3\": rpc error: code = NotFound desc = could not find container \"0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3\": container with ID starting with 0bcad62e9feef95445b11fe65aebacc2d6ae9806df8bcc343ee63ac775c599a3 not found: ID does not exist"
Sep 18 20:34:17.876953 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.876931    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"]
Sep 18 20:34:17.879748 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.879725    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58zpcz7"]
Sep 18 20:34:17.966918 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:17.966883    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8951194b-269a-496a-a12d-f39f8c093744" path="/var/lib/kubelet/pods/8951194b-269a-496a-a12d-f39f8c093744/volumes"
Sep 18 20:34:18.947146 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:18.947105    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:34:22.625103 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.625068    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"]
Sep 18 20:34:22.625748 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.625483    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main"
Sep 18 20:34:22.625748 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.625502    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main"
Sep 18 20:34:22.625748 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.625564    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8951194b-269a-496a-a12d-f39f8c093744" containerName="main"
Sep 18 20:34:22.628339 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.628316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.633577 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.633529    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 18 20:34:22.634081 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.634022    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-mlj7x\""
Sep 18 20:34:22.634217 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.634177    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 20:34:22.634305 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.634279    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 20:34:22.650604 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.650578    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"]
Sep 18 20:34:22.724875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.724844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.724885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.724911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725026 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.724981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725174 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.725024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725174 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.725054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725174 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.725088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwmd\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-kube-api-access-cqwmd\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725174 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.725149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.725302 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.725191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.825978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwmd\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-kube-api-access-cqwmd\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.826347 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.825995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.826456 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.826431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.826540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.826515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.826606 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.826587    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.826812 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.826787    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.827068 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.826801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.828550 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.828520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.828706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.828687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.836657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.836638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.836865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.836844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwmd\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-kube-api-access-cqwmd\") pod \"router-gateway-1-openshift-default-56bd58cf4d-prkr7\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:22.940010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:22.939936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:23.215751 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.215722    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"]
Sep 18 20:34:23.217682 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:34:23.217657    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c2f68e_a810_4f42_bd07_fa41586cb5b3.slice/crio-8b4b34fcf670b34ed6d2b7502ee2be1e535fd71b260cff3892ba0a2ae5a93194 WatchSource:0}: Error finding container 8b4b34fcf670b34ed6d2b7502ee2be1e535fd71b260cff3892ba0a2ae5a93194: Status 404 returned error can't find the container with id 8b4b34fcf670b34ed6d2b7502ee2be1e535fd71b260cff3892ba0a2ae5a93194
Sep 18 20:34:23.219753 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.219719    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:34:23.219844 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.219782    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:34:23.219844 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.219818    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:34:23.889230 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.889192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" event={"ID":"a9c2f68e-a810-4f42-bd07-fa41586cb5b3","Type":"ContainerStarted","Data":"baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2"}
Sep 18 20:34:23.889632 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.889244    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" event={"ID":"a9c2f68e-a810-4f42-bd07-fa41586cb5b3","Type":"ContainerStarted","Data":"8b4b34fcf670b34ed6d2b7502ee2be1e535fd71b260cff3892ba0a2ae5a93194"}
Sep 18 20:34:23.909282 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.909232    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" podStartSLOduration=1.9092157090000001 podStartE2EDuration="1.909215709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:23.908325099 +0000 UTC m=+3358.162531705" watchObservedRunningTime="2026-09-18 20:34:23.909215709 +0000 UTC m=+3358.163422315"
Sep 18 20:34:23.940858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:23.940835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:24.945680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:24.945652    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:25.897887 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:25.897839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:25.947787 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:25.947751    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:28.947172 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:28.947130    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:34:34.939040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:34.939000    2583 generic.go:356] "Generic (PLEG): container finished" podID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerID="baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2" exitCode=0
Sep 18 20:34:34.939481 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:34.939064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" event={"ID":"a9c2f68e-a810-4f42-bd07-fa41586cb5b3","Type":"ContainerDied","Data":"baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2"}
Sep 18 20:34:34.939566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:34.939549    2583 scope.go:117] "RemoveContainer" containerID="baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2"
Sep 18 20:34:34.940580 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:34.940550    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:34:34.940650 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:34.940634    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:34:34.940695 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:34.940680    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:34:35.947035 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:35.946989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" event={"ID":"a9c2f68e-a810-4f42-bd07-fa41586cb5b3","Type":"ContainerStarted","Data":"e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1"}
Sep 18 20:34:36.940624 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:36.940583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:36.942052 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:36.942002    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 20:34:37.940649 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:37.940601    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 20:34:38.940631 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:38.940569    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 20:34:38.948152 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:38.948101    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:34:39.941333 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:39.941279    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 20:34:40.898226 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:40.898182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:40.941295 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:40.941244    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 20:34:41.944461 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:41.944425    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:41.945595 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:41.945569    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:34:48.947949 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:48.947909    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:34:58.947436 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:34:58.947388    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 18 20:35:08.957485 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:08.957407    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:35:08.965804 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:08.965779    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:35:11.232843 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:11.232811    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"]
Sep 18 20:35:26.255348 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:26.255231    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" containerID="cri-o://fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604" gracePeriod=45
Sep 18 20:35:27.001877 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.001851    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:35:27.152713 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152685    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a053100-0efe-46e8-bc49-cc16e936376c-cabundle-cert\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.152890 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-home\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.152890 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152776    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-model-cache\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.152890 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-tmp-dir\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.153097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a053100-0efe-46e8-bc49-cc16e936376c-tls-certs\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.153097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152945    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-dshm\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.153097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzdb6\" (UniqueName: \"kubernetes.io/projected/7a053100-0efe-46e8-bc49-cc16e936376c-kube-api-access-bzdb6\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.153097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.152984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:27.153097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.153011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-kserve-provision-location\") pod \"7a053100-0efe-46e8-bc49-cc16e936376c\" (UID: \"7a053100-0efe-46e8-bc49-cc16e936376c\") "
Sep 18 20:35:27.153387 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.153188    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a053100-0efe-46e8-bc49-cc16e936376c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:27.153387 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.153253    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a053100-0efe-46e8-bc49-cc16e936376c-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.153387 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.153270    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.153582 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.153559    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-home" (OuterVolumeSpecName: "home") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:27.155433 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.155413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-dshm" (OuterVolumeSpecName: "dshm") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:27.155681 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.155649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a053100-0efe-46e8-bc49-cc16e936376c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:27.155681 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.155646    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a053100-0efe-46e8-bc49-cc16e936376c-kube-api-access-bzdb6" (OuterVolumeSpecName: "kube-api-access-bzdb6") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "kube-api-access-bzdb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:27.163706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.163676    2583 generic.go:356] "Generic (PLEG): container finished" podID="7a053100-0efe-46e8-bc49-cc16e936376c" containerID="fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604" exitCode=0
Sep 18 20:35:27.163816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.163720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" event={"ID":"7a053100-0efe-46e8-bc49-cc16e936376c","Type":"ContainerDied","Data":"fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604"}
Sep 18 20:35:27.163816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.163736    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"
Sep 18 20:35:27.163816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.163768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" event={"ID":"7a053100-0efe-46e8-bc49-cc16e936376c","Type":"ContainerDied","Data":"302d2587b2828f407c5c2d9faa9ddfe5dc274f24c8a6d19fe455164fb31eee8d"}
Sep 18 20:35:27.163816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.163790    2583 scope.go:117] "RemoveContainer" containerID="fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604"
Sep 18 20:35:27.166481 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.166452    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:27.171937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.171916    2583 scope.go:117] "RemoveContainer" containerID="a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d"
Sep 18 20:35:27.178692 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.178662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a053100-0efe-46e8-bc49-cc16e936376c" (UID: "7a053100-0efe-46e8-bc49-cc16e936376c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:27.179634 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.179610    2583 scope.go:117] "RemoveContainer" containerID="fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604"
Sep 18 20:35:27.179904 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:35:27.179886    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604\": container with ID starting with fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604 not found: ID does not exist" containerID="fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604"
Sep 18 20:35:27.179961 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.179912    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604"} err="failed to get container status \"fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604\": rpc error: code = NotFound desc = could not find container \"fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604\": container with ID starting with fdac04fec9f943b7ac38141ff6a9cedae000c342fa703318ea964bbf415e2604 not found: ID does not exist"
Sep 18 20:35:27.179961 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.179930    2583 scope.go:117] "RemoveContainer" containerID="a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d"
Sep 18 20:35:27.180188 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:35:27.180173    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d\": container with ID starting with a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d not found: ID does not exist" containerID="a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d"
Sep 18 20:35:27.180239 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.180191    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d"} err="failed to get container status \"a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d\": rpc error: code = NotFound desc = could not find container \"a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d\": container with ID starting with a609fe391582d0fdd5431246b530a9c18f57e92c6dcac41499aa571c2950db7d not found: ID does not exist"
Sep 18 20:35:27.254275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.254246    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.254275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.254270    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a053100-0efe-46e8-bc49-cc16e936376c-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.254275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.254280    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.254483 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.254289    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdb6\" (UniqueName: \"kubernetes.io/projected/7a053100-0efe-46e8-bc49-cc16e936376c-kube-api-access-bzdb6\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.254483 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.254298    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.254483 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.254306    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a053100-0efe-46e8-bc49-cc16e936376c-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:35:27.488814 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.488788    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"]
Sep 18 20:35:27.496722 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.496639    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt"]
Sep 18 20:35:27.958861 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.958824    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-dlrxt" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:35:27.967904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:27.967877    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" path="/var/lib/kubelet/pods/7a053100-0efe-46e8-bc49-cc16e936376c/volumes"
Sep 18 20:35:29.624114 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.624084    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"]
Sep 18 20:35:29.624548 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.624415    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main"
Sep 18 20:35:29.624548 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.624428    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main"
Sep 18 20:35:29.624548 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.624438    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="storage-initializer"
Sep 18 20:35:29.624548 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.624444    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="storage-initializer"
Sep 18 20:35:29.624548 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.624488    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a053100-0efe-46e8-bc49-cc16e936376c" containerName="main"
Sep 18 20:35:29.627038 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.627021    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.629592 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.629568    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 20:35:29.629592 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.629588    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 20:35:29.629831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.629668    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 20:35:29.629831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.629694    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:35:29.629831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.629729    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-g4sg4\""
Sep 18 20:35:29.629831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.629800    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 20:35:29.638561 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.638541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"]
Sep 18 20:35:29.773659 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrzg\" (UniqueName: \"kubernetes.io/projected/efa47864-d243-4298-86bd-35b00f34b01a-kube-api-access-xwrzg\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.773659 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-home\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.773868 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-dshm\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.773868 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-model-cache\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.773868 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-kserve-provision-location\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.773868 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa47864-d243-4298-86bd-35b00f34b01a-tls-certs\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.773868 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa47864-d243-4298-86bd-35b00f34b01a-cabundle-cert\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.773868 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.773861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-tmp-dir\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.840535 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.840502    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"]
Sep 18 20:35:29.843817 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.843796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:29.845957 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.845933    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-zglpv\""
Sep 18 20:35:29.853424 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.853400    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"]
Sep 18 20:35:29.874665 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.874599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrzg\" (UniqueName: \"kubernetes.io/projected/efa47864-d243-4298-86bd-35b00f34b01a-kube-api-access-xwrzg\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.874665 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.874636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-home\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.874955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.874665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-dshm\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.874955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.874846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-model-cache\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.874955 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.874886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-kserve-provision-location\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875146 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa47864-d243-4298-86bd-35b00f34b01a-tls-certs\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875146 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-home\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875146 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa47864-d243-4298-86bd-35b00f34b01a-cabundle-cert\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875306 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-tmp-dir\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875306 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-model-cache\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875306 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-kserve-provision-location\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-tmp-dir\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.875816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.875797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa47864-d243-4298-86bd-35b00f34b01a-cabundle-cert\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.877124 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.877101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-dshm\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.877525 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.877509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa47864-d243-4298-86bd-35b00f34b01a-tls-certs\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.882866 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.882843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrzg\" (UniqueName: \"kubernetes.io/projected/efa47864-d243-4298-86bd-35b00f34b01a-kube-api-access-xwrzg\") pod \"stop-feature-test-kserve-768994c8cb-wnvdj\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.937658 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.937635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:29.975778 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.975650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7259d1df-f0cd-456a-bb25-0b77432e20f0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:29.975778 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:29.975710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5m47\" (UniqueName: \"kubernetes.io/projected/7259d1df-f0cd-456a-bb25-0b77432e20f0-kube-api-access-w5m47\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:30.077029 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:30.076991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7259d1df-f0cd-456a-bb25-0b77432e20f0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:30.077208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:30.077035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5m47\" (UniqueName: \"kubernetes.io/projected/7259d1df-f0cd-456a-bb25-0b77432e20f0-kube-api-access-w5m47\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:30.079831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:30.079802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7259d1df-f0cd-456a-bb25-0b77432e20f0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:30.084626 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:30.084598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5m47\" (UniqueName: \"kubernetes.io/projected/7259d1df-f0cd-456a-bb25-0b77432e20f0-kube-api-access-w5m47\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:30.153594 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:30.153515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:30.213916 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:30.213882    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"]
Sep 18 20:35:30.215564 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:35:30.215529    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa47864_d243_4298_86bd_35b00f34b01a.slice/crio-60a3d57fd071cb30ddb67ceab3da786912ac4ab5ee47ba496c2e4e6666b72ef7 WatchSource:0}: Error finding container 60a3d57fd071cb30ddb67ceab3da786912ac4ab5ee47ba496c2e4e6666b72ef7: Status 404 returned error can't find the container with id 60a3d57fd071cb30ddb67ceab3da786912ac4ab5ee47ba496c2e4e6666b72ef7
Sep 18 20:35:30.422643 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:30.422611    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"]
Sep 18 20:35:30.425029 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:35:30.425002    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7259d1df_f0cd_456a_bb25_0b77432e20f0.slice/crio-65d4bc3bc0b112b8c030fc011f69c0f1c2fe6b3d2f91c17b956dc3e4d409d474 WatchSource:0}: Error finding container 65d4bc3bc0b112b8c030fc011f69c0f1c2fe6b3d2f91c17b956dc3e4d409d474: Status 404 returned error can't find the container with id 65d4bc3bc0b112b8c030fc011f69c0f1c2fe6b3d2f91c17b956dc3e4d409d474
Sep 18 20:35:31.181882 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:31.181846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" event={"ID":"7259d1df-f0cd-456a-bb25-0b77432e20f0","Type":"ContainerStarted","Data":"b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc"}
Sep 18 20:35:31.182384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:31.181901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" event={"ID":"7259d1df-f0cd-456a-bb25-0b77432e20f0","Type":"ContainerStarted","Data":"65d4bc3bc0b112b8c030fc011f69c0f1c2fe6b3d2f91c17b956dc3e4d409d474"}
Sep 18 20:35:31.182384 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:31.181940    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:35:31.183416 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:31.183391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" event={"ID":"efa47864-d243-4298-86bd-35b00f34b01a","Type":"ContainerStarted","Data":"6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1"}
Sep 18 20:35:31.183515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:31.183430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" event={"ID":"efa47864-d243-4298-86bd-35b00f34b01a","Type":"ContainerStarted","Data":"60a3d57fd071cb30ddb67ceab3da786912ac4ab5ee47ba496c2e4e6666b72ef7"}
Sep 18 20:35:31.198101 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:31.198058    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" podStartSLOduration=2.198043241 podStartE2EDuration="2.198043241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:35:31.197287389 +0000 UTC m=+3425.451493996" watchObservedRunningTime="2026-09-18 20:35:31.198043241 +0000 UTC m=+3425.452249846"
Sep 18 20:35:33.193471 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:33.193438    2583 generic.go:356] "Generic (PLEG): container finished" podID="efa47864-d243-4298-86bd-35b00f34b01a" containerID="6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1" exitCode=0
Sep 18 20:35:33.193872 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:33.193510    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" event={"ID":"efa47864-d243-4298-86bd-35b00f34b01a","Type":"ContainerDied","Data":"6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1"}
Sep 18 20:35:34.201189 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:34.201147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" event={"ID":"efa47864-d243-4298-86bd-35b00f34b01a","Type":"ContainerStarted","Data":"0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0"}
Sep 18 20:35:34.231674 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:34.231621    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podStartSLOduration=3.139102848 podStartE2EDuration="5.231604888s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.09250204s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:35:34.223439693 +0000 UTC m=+3428.477646297" watchObservedRunningTime="2026-09-18 20:35:34.231604888 +0000 UTC m=+3428.485811489"
Sep 18 20:35:34.937787 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:34.937748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:39.938786 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:39.938751    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:35:39.940452 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:39.940417    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:35:49.938274 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:49.938233    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:35:59.938747 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:35:59.938694    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:36:02.190459 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:36:02.190428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:36:09.938988 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:36:09.938940    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:36:19.938242 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:36:19.938202    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:36:29.938816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:36:29.938770    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:36:39.938766 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:36:39.938671    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:36:49.938786 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:36:49.938746    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:36:59.939055 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:36:59.939009    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:37:09.938778 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:09.938721    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:37:19.938914 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:19.938864    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:37:29.948509 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:29.948476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:37:29.956296 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:29.956269    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:37:31.562692 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:31.562656    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"]
Sep 18 20:37:31.575836 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:31.575807    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"]
Sep 18 20:37:32.190402 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:32.190349    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:37:42.190306 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:42.190262    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:37:46.583944 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:46.583898    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main" containerID="cri-o://b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc" gracePeriod=45
Sep 18 20:37:46.743233 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:46.743195    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" containerID="cri-o://0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0" gracePeriod=45
Sep 18 20:37:46.949704 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:46.949613    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 20:37:47.417915 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.417885    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:37:47.591031 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.590998    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa47864-d243-4298-86bd-35b00f34b01a-cabundle-cert\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591040    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-kserve-provision-location\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591067    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-model-cache\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa47864-d243-4298-86bd-35b00f34b01a-tls-certs\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591178    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwrzg\" (UniqueName: \"kubernetes.io/projected/efa47864-d243-4298-86bd-35b00f34b01a-kube-api-access-xwrzg\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-dshm\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-tmp-dir\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-home\") pod \"efa47864-d243-4298-86bd-35b00f34b01a\" (UID: \"efa47864-d243-4298-86bd-35b00f34b01a\") "
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591338    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-model-cache" (OuterVolumeSpecName: "model-cache") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:47.591540 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa47864-d243-4298-86bd-35b00f34b01a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:37:47.592028 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591596    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa47864-d243-4298-86bd-35b00f34b01a-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.592028 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.591612    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.592125 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.592045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-home" (OuterVolumeSpecName: "home") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:47.594014 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.593987    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-dshm" (OuterVolumeSpecName: "dshm") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:47.594242 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.594217    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa47864-d243-4298-86bd-35b00f34b01a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:37:47.594984 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.594958    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa47864-d243-4298-86bd-35b00f34b01a-kube-api-access-xwrzg" (OuterVolumeSpecName: "kube-api-access-xwrzg") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "kube-api-access-xwrzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:37:47.610021 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.609994    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:47.615166 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.615141    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efa47864-d243-4298-86bd-35b00f34b01a" (UID: "efa47864-d243-4298-86bd-35b00f34b01a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:47.692668 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.692645    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrzg\" (UniqueName: \"kubernetes.io/projected/efa47864-d243-4298-86bd-35b00f34b01a-kube-api-access-xwrzg\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.692668 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.692665    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.692799 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.692675    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.692799 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.692682    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.692799 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.692689    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa47864-d243-4298-86bd-35b00f34b01a-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.692799 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.692698    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa47864-d243-4298-86bd-35b00f34b01a-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:37:47.792839 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.792810    2583 generic.go:356] "Generic (PLEG): container finished" podID="efa47864-d243-4298-86bd-35b00f34b01a" containerID="0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0" exitCode=0
Sep 18 20:37:47.792944 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.792851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" event={"ID":"efa47864-d243-4298-86bd-35b00f34b01a","Type":"ContainerDied","Data":"0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0"}
Sep 18 20:37:47.792944 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.792873    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"
Sep 18 20:37:47.792944 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.792893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj" event={"ID":"efa47864-d243-4298-86bd-35b00f34b01a","Type":"ContainerDied","Data":"60a3d57fd071cb30ddb67ceab3da786912ac4ab5ee47ba496c2e4e6666b72ef7"}
Sep 18 20:37:47.792944 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.792915    2583 scope.go:117] "RemoveContainer" containerID="0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0"
Sep 18 20:37:47.801934 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.801917    2583 scope.go:117] "RemoveContainer" containerID="6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1"
Sep 18 20:37:47.809188 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.809169    2583 scope.go:117] "RemoveContainer" containerID="0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0"
Sep 18 20:37:47.809490 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:37:47.809473    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0\": container with ID starting with 0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0 not found: ID does not exist" containerID="0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0"
Sep 18 20:37:47.809541 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.809497    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0"} err="failed to get container status \"0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0\": rpc error: code = NotFound desc = could not find container \"0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0\": container with ID starting with 0daf93596914abb87a9263a98a7f5d3434739205ff53f07e13f7a95ec53b5fb0 not found: ID does not exist"
Sep 18 20:37:47.809541 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.809514    2583 scope.go:117] "RemoveContainer" containerID="6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1"
Sep 18 20:37:47.809771 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:37:47.809752    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1\": container with ID starting with 6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1 not found: ID does not exist" containerID="6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1"
Sep 18 20:37:47.809827 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.809780    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1"} err="failed to get container status \"6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1\": rpc error: code = NotFound desc = could not find container \"6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1\": container with ID starting with 6629f7887f7d76aad63abb98743fe8e64e4af88fe037527ee659d1390b3937e1 not found: ID does not exist"
Sep 18 20:37:47.814280 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.814257    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"]
Sep 18 20:37:47.818428 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.818405    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-768994c8cb-wnvdj"]
Sep 18 20:37:47.967812 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:47.967746    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa47864-d243-4298-86bd-35b00f34b01a" path="/var/lib/kubelet/pods/efa47864-d243-4298-86bd-35b00f34b01a/volumes"
Sep 18 20:37:52.190399 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:52.190329    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:37:52.190965 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:37:52.190557    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:38:02.190130 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:02.190094    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:38:12.190562 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:12.190465    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:38:16.826105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.826080    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:38:16.899663 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.899601    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7259d1df-f0cd-456a-bb25-0b77432e20f0-tls-certs\") pod \"7259d1df-f0cd-456a-bb25-0b77432e20f0\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") "
Sep 18 20:38:16.899781 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.899663    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5m47\" (UniqueName: \"kubernetes.io/projected/7259d1df-f0cd-456a-bb25-0b77432e20f0-kube-api-access-w5m47\") pod \"7259d1df-f0cd-456a-bb25-0b77432e20f0\" (UID: \"7259d1df-f0cd-456a-bb25-0b77432e20f0\") "
Sep 18 20:38:16.901807 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.901776    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7259d1df-f0cd-456a-bb25-0b77432e20f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7259d1df-f0cd-456a-bb25-0b77432e20f0" (UID: "7259d1df-f0cd-456a-bb25-0b77432e20f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:38:16.902015 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.901993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7259d1df-f0cd-456a-bb25-0b77432e20f0-kube-api-access-w5m47" (OuterVolumeSpecName: "kube-api-access-w5m47") pod "7259d1df-f0cd-456a-bb25-0b77432e20f0" (UID: "7259d1df-f0cd-456a-bb25-0b77432e20f0"). InnerVolumeSpecName "kube-api-access-w5m47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:38:16.915746 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.915720    2583 generic.go:356] "Generic (PLEG): container finished" podID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerID="b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc" exitCode=0
Sep 18 20:38:16.915863 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.915766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" event={"ID":"7259d1df-f0cd-456a-bb25-0b77432e20f0","Type":"ContainerDied","Data":"b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc"}
Sep 18 20:38:16.915863 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.915771    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"
Sep 18 20:38:16.915863 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.915805    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l" event={"ID":"7259d1df-f0cd-456a-bb25-0b77432e20f0","Type":"ContainerDied","Data":"65d4bc3bc0b112b8c030fc011f69c0f1c2fe6b3d2f91c17b956dc3e4d409d474"}
Sep 18 20:38:16.915863 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.915826    2583 scope.go:117] "RemoveContainer" containerID="b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc"
Sep 18 20:38:16.924803 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.924787    2583 scope.go:117] "RemoveContainer" containerID="b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc"
Sep 18 20:38:16.925069 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:38:16.925046    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc\": container with ID starting with b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc not found: ID does not exist" containerID="b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc"
Sep 18 20:38:16.925120 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.925078    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc"} err="failed to get container status \"b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc\": rpc error: code = NotFound desc = could not find container \"b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc\": container with ID starting with b373044af1339eaef5c4d571207aa2026fcbcf434f2030252b4a07e290168cdc not found: ID does not exist"
Sep 18 20:38:16.940174 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.940156    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"]
Sep 18 20:38:16.947228 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:16.947203    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gpq9l"]
Sep 18 20:38:17.000279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:17.000259    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7259d1df-f0cd-456a-bb25-0b77432e20f0-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:38:17.000389 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:17.000281    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5m47\" (UniqueName: \"kubernetes.io/projected/7259d1df-f0cd-456a-bb25-0b77432e20f0-kube-api-access-w5m47\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:38:17.965860 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:17.965829    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" path="/var/lib/kubelet/pods/7259d1df-f0cd-456a-bb25-0b77432e20f0/volumes"
Sep 18 20:38:28.130962 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:28.130933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:38:28.136199 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:38:28.136179    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:39:09.924060 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924027    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"]
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924326    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main"
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924338    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main"
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924366    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main"
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924372    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main"
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924382    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="storage-initializer"
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924388    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="storage-initializer"
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924441    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7259d1df-f0cd-456a-bb25-0b77432e20f0" containerName="main"
Sep 18 20:39:09.924507 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.924452    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa47864-d243-4298-86bd-35b00f34b01a" containerName="main"
Sep 18 20:39:09.927478 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.927457    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:09.929903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.929878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-9cgz8\""
Sep 18 20:39:09.930021 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.929878    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:39:09.930021 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.929925    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 18 20:39:09.930204 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.930187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 18 20:39:09.930343 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.930317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 18 20:39:09.930477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.930432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 18 20:39:09.940673 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:09.940651    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"]
Sep 18 20:39:10.083512 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tls-certs\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.083512 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-home\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.083778 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e52da1-7db8-41e9-a2d6-de2634d58e08-cabundle-cert\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.083778 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgnx\" (UniqueName: \"kubernetes.io/projected/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kube-api-access-vfgnx\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.083778 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tmp-dir\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.083778 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.084005 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-dshm\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.084005 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.083828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-model-cache\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184533 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tmp-dir\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184533 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184533 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-dshm\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184533 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-model-cache\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184533 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tls-certs\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184888 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-home\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184888 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e52da1-7db8-41e9-a2d6-de2634d58e08-cabundle-cert\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.184888 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgnx\" (UniqueName: \"kubernetes.io/projected/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kube-api-access-vfgnx\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.185059 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.184928    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tmp-dir\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.185059 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.185002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.185059 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.185023    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-model-cache\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.185499 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.185467    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e52da1-7db8-41e9-a2d6-de2634d58e08-cabundle-cert\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.185499 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.185496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-home\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.187011 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.186989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-dshm\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.187179 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.187156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tls-certs\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.192278 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.192255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgnx\" (UniqueName: \"kubernetes.io/projected/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kube-api-access-vfgnx\") pod \"lora-single-adapter-test-kserve-58dd6f88c4-ttqjq\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.238110 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.238087    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:10.517291 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:10.517249    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"]
Sep 18 20:39:10.521067 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:39:10.521039    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e52da1_7db8_41e9_a2d6_de2634d58e08.slice/crio-cc920447616954f413b67f720727f7f2828fecca3645dc290192a4b2d238307a WatchSource:0}: Error finding container cc920447616954f413b67f720727f7f2828fecca3645dc290192a4b2d238307a: Status 404 returned error can't find the container with id cc920447616954f413b67f720727f7f2828fecca3645dc290192a4b2d238307a
Sep 18 20:39:11.138440 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:11.138401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" event={"ID":"a6e52da1-7db8-41e9-a2d6-de2634d58e08","Type":"ContainerStarted","Data":"8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7"}
Sep 18 20:39:11.138894 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:11.138486    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" event={"ID":"a6e52da1-7db8-41e9-a2d6-de2634d58e08","Type":"ContainerStarted","Data":"cc920447616954f413b67f720727f7f2828fecca3645dc290192a4b2d238307a"}
Sep 18 20:39:14.152669 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:14.152635    2583 generic.go:356] "Generic (PLEG): container finished" podID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerID="8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7" exitCode=0
Sep 18 20:39:14.153046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:14.152707    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" event={"ID":"a6e52da1-7db8-41e9-a2d6-de2634d58e08","Type":"ContainerDied","Data":"8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7"}
Sep 18 20:39:15.159225 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:15.159194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" event={"ID":"a6e52da1-7db8-41e9-a2d6-de2634d58e08","Type":"ContainerStarted","Data":"7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc"}
Sep 18 20:39:15.183398 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:15.183333    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podStartSLOduration=3.42455081 podStartE2EDuration="6.183319201s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.758768391s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:39:15.181617313 +0000 UTC m=+3649.435823920" watchObservedRunningTime="2026-09-18 20:39:15.183319201 +0000 UTC m=+3649.437525804"
Sep 18 20:39:15.239141 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:15.239100    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:20.239275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:20.239232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:39:20.240903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:20.240871    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:39:30.239033 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:30.238989    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:39:40.239065 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:40.238971    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:39:50.239189 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:39:50.239140    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:40:00.238807 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:00.238759    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:40:10.238853 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:10.238816    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:40:20.239232 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:20.239187    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:40:30.249010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:30.248973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:40:30.256799 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:30.256772    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:40:54.418334 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:54.418302    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-58dd6f88c4-ttqjq_a6e52da1-7db8-41e9-a2d6-de2634d58e08/storage-initializer/0.log"
Sep 18 20:40:54.427245 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:54.427220    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-58dd6f88c4-ttqjq_a6e52da1-7db8-41e9-a2d6-de2634d58e08/main/0.log"
Sep 18 20:40:55.684759 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:40:55.684725    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"]
Sep 18 20:41:10.708750 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:10.708664    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" containerID="cri-o://7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc" gracePeriod=45
Sep 18 20:41:11.250635 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.250597    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 20:41:11.356568 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.356542    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:41:11.395388 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395310    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kserve-provision-location\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395388 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tmp-dir\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395577 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-home\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395577 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-dshm\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395577 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-model-cache\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395577 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfgnx\" (UniqueName: \"kubernetes.io/projected/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kube-api-access-vfgnx\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395577 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e52da1-7db8-41e9-a2d6-de2634d58e08-cabundle-cert\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395845 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395597    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tls-certs\") pod \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\" (UID: \"a6e52da1-7db8-41e9-a2d6-de2634d58e08\") "
Sep 18 20:41:11.395845 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395752    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:11.395845 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395823    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-home" (OuterVolumeSpecName: "home") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:11.396013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.395843    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.396228 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.396206    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e52da1-7db8-41e9-a2d6-de2634d58e08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:11.398134 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.398107    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:41:11.398244 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.398225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-dshm" (OuterVolumeSpecName: "dshm") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:11.398368 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.398333    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kube-api-access-vfgnx" (OuterVolumeSpecName: "kube-api-access-vfgnx") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "kube-api-access-vfgnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:11.407517 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.407487    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:11.424255 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.424230    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6e52da1-7db8-41e9-a2d6-de2634d58e08" (UID: "a6e52da1-7db8-41e9-a2d6-de2634d58e08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:11.496419 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.496391    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgnx\" (UniqueName: \"kubernetes.io/projected/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kube-api-access-vfgnx\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.496419 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.496419    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6e52da1-7db8-41e9-a2d6-de2634d58e08-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.496538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.496429    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.496538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.496437    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.496538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.496445    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.496538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.496453    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.496538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.496459    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e52da1-7db8-41e9-a2d6-de2634d58e08-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.661996 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.661923    2583 generic.go:356] "Generic (PLEG): container finished" podID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerID="7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc" exitCode=0
Sep 18 20:41:11.661996 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.661960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" event={"ID":"a6e52da1-7db8-41e9-a2d6-de2634d58e08","Type":"ContainerDied","Data":"7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc"}
Sep 18 20:41:11.661996 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.661978    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"
Sep 18 20:41:11.662202 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.661998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq" event={"ID":"a6e52da1-7db8-41e9-a2d6-de2634d58e08","Type":"ContainerDied","Data":"cc920447616954f413b67f720727f7f2828fecca3645dc290192a4b2d238307a"}
Sep 18 20:41:11.662202 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.662020    2583 scope.go:117] "RemoveContainer" containerID="7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc"
Sep 18 20:41:11.671301 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.671284    2583 scope.go:117] "RemoveContainer" containerID="8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7"
Sep 18 20:41:11.679222 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.679203    2583 scope.go:117] "RemoveContainer" containerID="7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc"
Sep 18 20:41:11.679504 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:41:11.679485    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc\": container with ID starting with 7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc not found: ID does not exist" containerID="7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc"
Sep 18 20:41:11.679585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.679515    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc"} err="failed to get container status \"7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc\": rpc error: code = NotFound desc = could not find container \"7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc\": container with ID starting with 7b944e0f0aeeee00301c86f40dca217a0aa9e2e50d8ce7a5a0a784e313f4bacc not found: ID does not exist"
Sep 18 20:41:11.679585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.679539    2583 scope.go:117] "RemoveContainer" containerID="8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7"
Sep 18 20:41:11.679822 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:41:11.679804    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7\": container with ID starting with 8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7 not found: ID does not exist" containerID="8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7"
Sep 18 20:41:11.679866 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.679830    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7"} err="failed to get container status \"8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7\": rpc error: code = NotFound desc = could not find container \"8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7\": container with ID starting with 8fd94230aaabe1b278be42cc9f3c3b642cba8d4641353ba444d719f1d7a03db7 not found: ID does not exist"
Sep 18 20:41:11.684206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.684188    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"]
Sep 18 20:41:11.688413 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.688393    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-58dd6f88c4-ttqjq"]
Sep 18 20:41:11.966991 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:11.966888    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" path="/var/lib/kubelet/pods/a6e52da1-7db8-41e9-a2d6-de2634d58e08/volumes"
Sep 18 20:41:31.201820 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:31.201782    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"]
Sep 18 20:41:31.202234 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:31.201976    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy" containerID="cri-o://e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1" gracePeriod=30
Sep 18 20:41:36.453659 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.453634    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:41:36.577623 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-data\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.577768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-envoy\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.577768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577666    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqwmd\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-kube-api-access-cqwmd\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.577768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-podinfo\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.577768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577705    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-credential-socket\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.577768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istiod-ca-cert\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.577768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-socket\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.578096 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577793    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-token\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.578096 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577814    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-certs\") pod \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\" (UID: \"a9c2f68e-a810-4f42-bd07-fa41586cb5b3\") "
Sep 18 20:41:36.578096 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.577892    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:36.578096 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.578029    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-data\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.578096 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.578032    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:36.578553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.578093    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:36.578553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.578321    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:36.578553 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.578409    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:36.580230 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.580199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-kube-api-access-cqwmd" (OuterVolumeSpecName: "kube-api-access-cqwmd") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "kube-api-access-cqwmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:36.580230 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.580210    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:41:36.580230 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.580221    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:36.580416 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.580209    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "a9c2f68e-a810-4f42-bd07-fa41586cb5b3" (UID: "a9c2f68e-a810-4f42-bd07-fa41586cb5b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:36.678688 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678661    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-podinfo\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.678688 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678683    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-credential-socket\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.678826 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678692    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istiod-ca-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.678826 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678700    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-socket\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.678826 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678708    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-token\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.678826 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678715    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-workload-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.678826 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678723    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-istio-envoy\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.678826 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.678731    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwmd\" (UniqueName: \"kubernetes.io/projected/a9c2f68e-a810-4f42-bd07-fa41586cb5b3-kube-api-access-cqwmd\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:41:36.763677 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.763649    2583 generic.go:356] "Generic (PLEG): container finished" podID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerID="e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1" exitCode=0
Sep 18 20:41:36.763783 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.763697    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"
Sep 18 20:41:36.763783 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.763733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" event={"ID":"a9c2f68e-a810-4f42-bd07-fa41586cb5b3","Type":"ContainerDied","Data":"e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1"}
Sep 18 20:41:36.763863 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.763787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7" event={"ID":"a9c2f68e-a810-4f42-bd07-fa41586cb5b3","Type":"ContainerDied","Data":"8b4b34fcf670b34ed6d2b7502ee2be1e535fd71b260cff3892ba0a2ae5a93194"}
Sep 18 20:41:36.763863 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.763816    2583 scope.go:117] "RemoveContainer" containerID="e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1"
Sep 18 20:41:36.772654 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.772633    2583 scope.go:117] "RemoveContainer" containerID="baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2"
Sep 18 20:41:36.780521 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.780504    2583 scope.go:117] "RemoveContainer" containerID="e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1"
Sep 18 20:41:36.780792 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:41:36.780773    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1\": container with ID starting with e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1 not found: ID does not exist" containerID="e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1"
Sep 18 20:41:36.780831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.780802    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1"} err="failed to get container status \"e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1\": rpc error: code = NotFound desc = could not find container \"e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1\": container with ID starting with e912347a7d9e7ad0593d07c3fcc87cf57951290d63188a5956c563e01440a5d1 not found: ID does not exist"
Sep 18 20:41:36.780831 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.780818    2583 scope.go:117] "RemoveContainer" containerID="baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2"
Sep 18 20:41:36.781070 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:41:36.781053    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2\": container with ID starting with baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2 not found: ID does not exist" containerID="baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2"
Sep 18 20:41:36.781107 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.781076    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2"} err="failed to get container status \"baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2\": rpc error: code = NotFound desc = could not find container \"baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2\": container with ID starting with baf648b6ffd79a77d78f1cd5ebcf769b3b76c71fe5aa506f940241b0ceaab2c2 not found: ID does not exist"
Sep 18 20:41:36.785723 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.785703    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"]
Sep 18 20:41:36.790299 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:36.790280    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-prkr7"]
Sep 18 20:41:37.965286 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:37.965252    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" path="/var/lib/kubelet/pods/a9c2f68e-a810-4f42-bd07-fa41586cb5b3/volumes"
Sep 18 20:41:42.040823 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.040789    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"]
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041034    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041044    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041053    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041060    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041076    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="storage-initializer"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041082    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="storage-initializer"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041125    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041132    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041138    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e52da1-7db8-41e9-a2d6-de2634d58e08" containerName="main"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041175    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy"
Sep 18 20:41:42.041203 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.041181    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c2f68e-a810-4f42-bd07-fa41586cb5b3" containerName="istio-proxy"
Sep 18 20:41:42.045913 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.045889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.051244 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.051215    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 20:41:42.051404 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.051227    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-smjj5\""
Sep 18 20:41:42.051680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.051661    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 20:41:42.051774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.051673    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-jqwrs\""
Sep 18 20:41:42.051774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.051668    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 20:41:42.051774 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.051671    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:41:42.052208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.052188    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 18 20:41:42.070002 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.069979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"]
Sep 18 20:41:42.117253 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.117398 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.117398 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48529211-be1d-47d1-b07c-8f5bede93241-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.117398 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tqs\" (UniqueName: \"kubernetes.io/projected/48529211-be1d-47d1-b07c-8f5bede93241-kube-api-access-69tqs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.117538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.117538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.117538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.117538 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.117462    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48529211-be1d-47d1-b07c-8f5bede93241-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218236 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48529211-be1d-47d1-b07c-8f5bede93241-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218236 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tqs\" (UniqueName: \"kubernetes.io/projected/48529211-be1d-47d1-b07c-8f5bede93241-kube-api-access-69tqs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218473 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218473 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218473 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218473 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48529211-be1d-47d1-b07c-8f5bede93241-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218696 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218658    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218752 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218818 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.218915 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.219002 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.218982    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.219083 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.219064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48529211-be1d-47d1-b07c-8f5bede93241-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.219319 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.219185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.220869 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.220848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.221131 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.221113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48529211-be1d-47d1-b07c-8f5bede93241-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.225855 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.225832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tqs\" (UniqueName: \"kubernetes.io/projected/48529211-be1d-47d1-b07c-8f5bede93241-kube-api-access-69tqs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.356175 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.356107    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:42.637680 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.637587    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"]
Sep 18 20:41:42.641317 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:41:42.641287    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48529211_be1d_47d1_b07c_8f5bede93241.slice/crio-d2c5ca948216265434d7bfc77ed5ed677d86cd1888ba4c011a61f0e44bfa2d15 WatchSource:0}: Error finding container d2c5ca948216265434d7bfc77ed5ed677d86cd1888ba4c011a61f0e44bfa2d15: Status 404 returned error can't find the container with id d2c5ca948216265434d7bfc77ed5ed677d86cd1888ba4c011a61f0e44bfa2d15
Sep 18 20:41:42.643277 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.643260    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:41:42.791010 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:42.790979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerStarted","Data":"d2c5ca948216265434d7bfc77ed5ed677d86cd1888ba4c011a61f0e44bfa2d15"}
Sep 18 20:41:44.801367 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:44.801313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerStarted","Data":"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"}
Sep 18 20:41:44.801768 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:44.801443    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:45.807647 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:45.807587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerStarted","Data":"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"}
Sep 18 20:41:47.817879 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:47.817842    2583 generic.go:356] "Generic (PLEG): container finished" podID="48529211-be1d-47d1-b07c-8f5bede93241" containerID="74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492" exitCode=0
Sep 18 20:41:47.818402 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:47.817924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerDied","Data":"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"}
Sep 18 20:41:48.823786 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:48.823740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerStarted","Data":"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"}
Sep 18 20:41:48.850678 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:48.850627    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podStartSLOduration=3.611459231 podStartE2EDuration="6.850607394s" totalImagesPullingTime="1.136721485s" totalInitContainerRuntime="2.102426678s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:48.848499518 +0000 UTC m=+3803.102706122" watchObservedRunningTime="2026-09-18 20:41:48.850607394 +0000 UTC m=+3803.104813999"
Sep 18 20:41:52.356929 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:52.356892    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:52.357331 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:52.356969    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:41:52.358218 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:41:52.358191    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:42:02.357368 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:42:02.357314    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:42:02.370588 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:42:02.370564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:42:12.356719 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:42:12.356673    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:42:22.357537 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:42:22.357481    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:42:32.356534 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:42:32.356482    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:42:42.356767 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:42:42.356647    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:42:52.357206 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:42:52.357153    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:43:02.357220 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:02.357156    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:43:12.357421 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:12.357347    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:43:22.356777 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:22.356727    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:43:28.158373 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:28.158333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:43:28.164994 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:28.164966    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:43:32.356784 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:32.356732    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8001/health\": dial tcp 10.134.0.61:8001: connect: connection refused"
Sep 18 20:43:42.367092 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:42.367053    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:43:42.379105 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:43:42.379082    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:44:21.556609 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:21.556576    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"]
Sep 18 20:44:36.579731 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:36.579674    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main" containerID="cri-o://bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642" gracePeriod=45
Sep 18 20:44:37.095410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.095380    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="llm-d-routing-sidecar" containerID="cri-o://9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c" gracePeriod=45
Sep 18 20:44:37.335012 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.334990    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:44:37.402577 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402546    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-dshm\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.402716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402587    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-home\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.402716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-tmp-dir\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.402716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-kserve-provision-location\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.402716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402658    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69tqs\" (UniqueName: \"kubernetes.io/projected/48529211-be1d-47d1-b07c-8f5bede93241-kube-api-access-69tqs\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.402716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402674    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48529211-be1d-47d1-b07c-8f5bede93241-cabundle-cert\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.402716 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48529211-be1d-47d1-b07c-8f5bede93241-tls-certs\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.403046 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.402725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-model-cache\") pod \"48529211-be1d-47d1-b07c-8f5bede93241\" (UID: \"48529211-be1d-47d1-b07c-8f5bede93241\") "
Sep 18 20:44:37.403132 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.403101    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48529211-be1d-47d1-b07c-8f5bede93241-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:37.403308 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.403287    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-model-cache" (OuterVolumeSpecName: "model-cache") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.403496 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.403478    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48529211-be1d-47d1-b07c-8f5bede93241-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.403614 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.403585    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-home" (OuterVolumeSpecName: "home") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.405108 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.405081    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48529211-be1d-47d1-b07c-8f5bede93241-kube-api-access-69tqs" (OuterVolumeSpecName: "kube-api-access-69tqs") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "kube-api-access-69tqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:44:37.405343 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.405322    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-dshm" (OuterVolumeSpecName: "dshm") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.405442 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.405409    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48529211-be1d-47d1-b07c-8f5bede93241-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:44:37.416657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.416634    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.427493 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.427469    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48529211-be1d-47d1-b07c-8f5bede93241" (UID: "48529211-be1d-47d1-b07c-8f5bede93241"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.504842 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.504814    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69tqs\" (UniqueName: \"kubernetes.io/projected/48529211-be1d-47d1-b07c-8f5bede93241-kube-api-access-69tqs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.504842 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.504840    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48529211-be1d-47d1-b07c-8f5bede93241-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.504987 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.504855    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.504987 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.504866    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.504987 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.504877    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.504987 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.504888    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.504987 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.504899    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48529211-be1d-47d1-b07c-8f5bede93241-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.557610 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.557582    2583 generic.go:356] "Generic (PLEG): container finished" podID="48529211-be1d-47d1-b07c-8f5bede93241" containerID="bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642" exitCode=0
Sep 18 20:44:37.557610 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.557611    2583 generic.go:356] "Generic (PLEG): container finished" podID="48529211-be1d-47d1-b07c-8f5bede93241" containerID="9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c" exitCode=0
Sep 18 20:44:37.557790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.557642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerDied","Data":"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"}
Sep 18 20:44:37.557790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.557651    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"
Sep 18 20:44:37.557790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.557729    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerDied","Data":"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"}
Sep 18 20:44:37.557790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.557758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5" event={"ID":"48529211-be1d-47d1-b07c-8f5bede93241","Type":"ContainerDied","Data":"d2c5ca948216265434d7bfc77ed5ed677d86cd1888ba4c011a61f0e44bfa2d15"}
Sep 18 20:44:37.557790 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.557788    2583 scope.go:117] "RemoveContainer" containerID="bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"
Sep 18 20:44:37.565847 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.565817    2583 scope.go:117] "RemoveContainer" containerID="74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"
Sep 18 20:44:37.573492 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.573477    2583 scope.go:117] "RemoveContainer" containerID="9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"
Sep 18 20:44:37.580255 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.580235    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"]
Sep 18 20:44:37.581864 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.581777    2583 scope.go:117] "RemoveContainer" containerID="bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"
Sep 18 20:44:37.582056 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:44:37.582037    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642\": container with ID starting with bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642 not found: ID does not exist" containerID="bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"
Sep 18 20:44:37.582104 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582065    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"} err="failed to get container status \"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642\": rpc error: code = NotFound desc = could not find container \"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642\": container with ID starting with bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642 not found: ID does not exist"
Sep 18 20:44:37.582104 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582081    2583 scope.go:117] "RemoveContainer" containerID="74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"
Sep 18 20:44:37.582297 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:44:37.582282    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492\": container with ID starting with 74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492 not found: ID does not exist" containerID="74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"
Sep 18 20:44:37.582339 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582301    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"} err="failed to get container status \"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492\": rpc error: code = NotFound desc = could not find container \"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492\": container with ID starting with 74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492 not found: ID does not exist"
Sep 18 20:44:37.582339 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582313    2583 scope.go:117] "RemoveContainer" containerID="9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"
Sep 18 20:44:37.582587 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:44:37.582570    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c\": container with ID starting with 9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c not found: ID does not exist" containerID="9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"
Sep 18 20:44:37.582625 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582592    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"} err="failed to get container status \"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c\": rpc error: code = NotFound desc = could not find container \"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c\": container with ID starting with 9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c not found: ID does not exist"
Sep 18 20:44:37.582625 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582613    2583 scope.go:117] "RemoveContainer" containerID="bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"
Sep 18 20:44:37.582845 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582827    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642"} err="failed to get container status \"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642\": rpc error: code = NotFound desc = could not find container \"bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642\": container with ID starting with bb3ac8699d0763a668239a4414266f01c52e77c3ee32ad5f7d622a6a45e40642 not found: ID does not exist"
Sep 18 20:44:37.582909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.582845    2583 scope.go:117] "RemoveContainer" containerID="74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"
Sep 18 20:44:37.583107 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.583092    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492"} err="failed to get container status \"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492\": rpc error: code = NotFound desc = could not find container \"74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492\": container with ID starting with 74da0d22ed84bb22b8fe49f4e1063bfc427ca6886deeea5d7e0f93a9b2d1a492 not found: ID does not exist"
Sep 18 20:44:37.583153 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.583109    2583 scope.go:117] "RemoveContainer" containerID="9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"
Sep 18 20:44:37.583335 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.583318    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c"} err="failed to get container status \"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c\": rpc error: code = NotFound desc = could not find container \"9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c\": container with ID starting with 9ed283893645fb6c329ea397110c44bd6fbf55c80d6980593368eaf7a182e12c not found: ID does not exist"
Sep 18 20:44:37.587107 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.587084    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bfc5fdf5d-rqfx5"]
Sep 18 20:44:37.966817 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:44:37.966745    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48529211-be1d-47d1-b07c-8f5bede93241" path="/var/lib/kubelet/pods/48529211-be1d-47d1-b07c-8f5bede93241/volumes"
Sep 18 20:45:24.477040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477008    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"]
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477289    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="llm-d-routing-sidecar"
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477301    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="llm-d-routing-sidecar"
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477312    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main"
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477318    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main"
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477327    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="storage-initializer"
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477337    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="storage-initializer"
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477402    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="main"
Sep 18 20:45:24.477524 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.477413    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48529211-be1d-47d1-b07c-8f5bede93241" containerName="llm-d-routing-sidecar"
Sep 18 20:45:24.479437 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.479420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.482245 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.482217    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 20:45:24.482421 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.482296    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-4t7lw\""
Sep 18 20:45:24.482734 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.482708    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 20:45:24.482813 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.482767    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 20:45:24.482813 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.482796    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:45:24.483348 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.483320    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 20:45:24.483348 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.483326    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zt8ws\""
Sep 18 20:45:24.503658 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.503632    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"]
Sep 18 20:45:24.553222 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.553339 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-home\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.553339 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d629e2a9-e63b-4113-90ac-93363c1733ab-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.553446 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.553446 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkprj\" (UniqueName: \"kubernetes.io/projected/d629e2a9-e63b-4113-90ac-93363c1733ab-kube-api-access-qkprj\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.553446 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.553548 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-dshm\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.553548 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.553488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d629e2a9-e63b-4113-90ac-93363c1733ab-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654542 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkprj\" (UniqueName: \"kubernetes.io/projected/d629e2a9-e63b-4113-90ac-93363c1733ab-kube-api-access-qkprj\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-dshm\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d629e2a9-e63b-4113-90ac-93363c1733ab-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654657 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-home\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d629e2a9-e63b-4113-90ac-93363c1733ab-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654875 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.654975 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.654955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.655266 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.655244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-model-cache\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.655266 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.655277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.655643 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.655448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d629e2a9-e63b-4113-90ac-93363c1733ab-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.655643 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.655539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-home\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.657071 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.657051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-dshm\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.657212 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.657195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d629e2a9-e63b-4113-90ac-93363c1733ab-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.663211 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.663189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkprj\" (UniqueName: \"kubernetes.io/projected/d629e2a9-e63b-4113-90ac-93363c1733ab-kube-api-access-qkprj\") pod \"custom-route-timeout-pd-test-kserve-66858595f7-cjsvf\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:24.790712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:24.790680    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:25.065529 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:25.065455    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"]
Sep 18 20:45:25.069123 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:45:25.069096    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd629e2a9_e63b_4113_90ac_93363c1733ab.slice/crio-26b58e2890f640031d698975e9fef2e556e968862633e80b00a0dfd59f9a0bdb WatchSource:0}: Error finding container 26b58e2890f640031d698975e9fef2e556e968862633e80b00a0dfd59f9a0bdb: Status 404 returned error can't find the container with id 26b58e2890f640031d698975e9fef2e556e968862633e80b00a0dfd59f9a0bdb
Sep 18 20:45:25.746020 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:25.745984    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerStarted","Data":"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"}
Sep 18 20:45:25.746455 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:25.746038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerStarted","Data":"26b58e2890f640031d698975e9fef2e556e968862633e80b00a0dfd59f9a0bdb"}
Sep 18 20:45:25.746455 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:25.746124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:26.755064 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:26.755013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerStarted","Data":"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"}
Sep 18 20:45:28.765500 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:28.765469    2583 generic.go:356] "Generic (PLEG): container finished" podID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerID="4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4" exitCode=0
Sep 18 20:45:28.765904 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:28.765542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerDied","Data":"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"}
Sep 18 20:45:29.775644 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:29.775606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerStarted","Data":"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"}
Sep 18 20:45:29.799224 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:29.799137    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podStartSLOduration=3.692112546 podStartE2EDuration="5.799112534s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.106999988s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:45:29.797325763 +0000 UTC m=+4024.051532378" watchObservedRunningTime="2026-09-18 20:45:29.799112534 +0000 UTC m=+4024.053319139"
Sep 18 20:45:34.791114 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:34.791075    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:34.791617 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:34.791165    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:34.792475 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:34.792438    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:45:44.791454 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:44.791341    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:45:44.803905 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:44.803878    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:45:54.791537 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:45:54.791477    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:46:04.791896 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:46:04.791852    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:46:14.791993 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:46:14.791943    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:46:24.791960 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:46:24.791906    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:46:34.791762 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:46:34.791712    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:46:44.791298 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:46:44.791249    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:46:54.791637 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:46:54.791576    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:47:04.791068 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:04.791023    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:47:14.791759 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:14.791655    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Sep 18 20:47:19.740477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.740446    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"]
Sep 18 20:47:19.746504 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.746480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.750972 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.750949    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 20:47:19.751119 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.750967    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 20:47:19.751119 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.750949    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 20:47:19.751119 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.751090    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:47:19.751272 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.751116    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-gfj9n\""
Sep 18 20:47:19.751272 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.751231    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 20:47:19.761207 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.761183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"]
Sep 18 20:47:19.876252 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30be37cb-987d-4c79-8c91-798963a2885d-tls-certs\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.876482 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30be37cb-987d-4c79-8c91-798963a2885d-cabundle-cert\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.876482 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-dshm\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.876482 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-model-cache\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.876482 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-kserve-provision-location\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.876687 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-home\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.876687 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-tmp-dir\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.876687 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.876545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5gc\" (UniqueName: \"kubernetes.io/projected/30be37cb-987d-4c79-8c91-798963a2885d-kube-api-access-4s5gc\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.977681 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-model-cache\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.977862 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-kserve-provision-location\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.977862 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-home\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.977862 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-tmp-dir\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.977862 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5gc\" (UniqueName: \"kubernetes.io/projected/30be37cb-987d-4c79-8c91-798963a2885d-kube-api-access-4s5gc\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.977862 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30be37cb-987d-4c79-8c91-798963a2885d-tls-certs\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.978134 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30be37cb-987d-4c79-8c91-798963a2885d-cabundle-cert\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.978134 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.977898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-dshm\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.978237 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.978153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-model-cache\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.978564 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.978536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-kserve-provision-location\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.978685 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.978575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-home\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.978685 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.978609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-tmp-dir\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.979048 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.979024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30be37cb-987d-4c79-8c91-798963a2885d-cabundle-cert\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.980913 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.980885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-dshm\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.981397 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.981376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30be37cb-987d-4c79-8c91-798963a2885d-tls-certs\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:19.990228 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:19.990203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5gc\" (UniqueName: \"kubernetes.io/projected/30be37cb-987d-4c79-8c91-798963a2885d-kube-api-access-4s5gc\") pod \"prestop-hook-test-kserve-787cdcb6df-r9z9m\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:20.056325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:20.056296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:20.336598 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:20.336513    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"]
Sep 18 20:47:20.339843 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:47:20.339812    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30be37cb_987d_4c79_8c91_798963a2885d.slice/crio-39ea48db6b39f782b988ef8b5dfbe46c6548f977a970123a4ba7e1f3864e4195 WatchSource:0}: Error finding container 39ea48db6b39f782b988ef8b5dfbe46c6548f977a970123a4ba7e1f3864e4195: Status 404 returned error can't find the container with id 39ea48db6b39f782b988ef8b5dfbe46c6548f977a970123a4ba7e1f3864e4195
Sep 18 20:47:21.255143 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:21.255090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" event={"ID":"30be37cb-987d-4c79-8c91-798963a2885d","Type":"ContainerStarted","Data":"84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2"}
Sep 18 20:47:21.255662 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:21.255165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" event={"ID":"30be37cb-987d-4c79-8c91-798963a2885d","Type":"ContainerStarted","Data":"39ea48db6b39f782b988ef8b5dfbe46c6548f977a970123a4ba7e1f3864e4195"}
Sep 18 20:47:23.265623 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:23.265589    2583 generic.go:356] "Generic (PLEG): container finished" podID="30be37cb-987d-4c79-8c91-798963a2885d" containerID="84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2" exitCode=0
Sep 18 20:47:23.266023 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:23.265656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" event={"ID":"30be37cb-987d-4c79-8c91-798963a2885d","Type":"ContainerDied","Data":"84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2"}
Sep 18 20:47:24.272427 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:24.272387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" event={"ID":"30be37cb-987d-4c79-8c91-798963a2885d","Type":"ContainerStarted","Data":"f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e"}
Sep 18 20:47:24.294090 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:24.294041    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podStartSLOduration=3.157809952 podStartE2EDuration="5.29402257s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.136212618s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:47:24.293548029 +0000 UTC m=+4138.547754633" watchObservedRunningTime="2026-09-18 20:47:24.29402257 +0000 UTC m=+4138.548229174"
Sep 18 20:47:24.800624 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:24.800594    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:47:24.818769 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:24.818741    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:47:25.056587 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:25.056490    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:30.056805 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:30.056769    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:47:30.058262 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:30.058230    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:47:39.281551 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:39.281510    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"]
Sep 18 20:47:40.057630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:40.057585    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:47:50.057591 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:50.057539    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:47:54.309161 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:54.309119    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main" containerID="cri-o://5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4" gracePeriod=45
Sep 18 20:47:54.922308 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:54.922275    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="llm-d-routing-sidecar" containerID="cri-o://04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004" gracePeriod=45
Sep 18 20:47:55.164758 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.164728    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:47:55.273898 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.273866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-tmp-dir\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.273898 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.273901    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkprj\" (UniqueName: \"kubernetes.io/projected/d629e2a9-e63b-4113-90ac-93363c1733ab-kube-api-access-qkprj\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.274147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.273919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-dshm\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.274147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.273940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d629e2a9-e63b-4113-90ac-93363c1733ab-cabundle-cert\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.274147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.273984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-model-cache\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.274147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.274045    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-home\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.274147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.274077    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-kserve-provision-location\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.274147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.274100    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d629e2a9-e63b-4113-90ac-93363c1733ab-tls-certs\") pod \"d629e2a9-e63b-4113-90ac-93363c1733ab\" (UID: \"d629e2a9-e63b-4113-90ac-93363c1733ab\") "
Sep 18 20:47:55.274485 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.274280    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.274485 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.274452    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d629e2a9-e63b-4113-90ac-93363c1733ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:55.274767 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.274733    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-home" (OuterVolumeSpecName: "home") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.276492 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.276468    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d629e2a9-e63b-4113-90ac-93363c1733ab-kube-api-access-qkprj" (OuterVolumeSpecName: "kube-api-access-qkprj") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "kube-api-access-qkprj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:47:55.276997 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.276968    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d629e2a9-e63b-4113-90ac-93363c1733ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:47:55.277121 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.277104    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-dshm" (OuterVolumeSpecName: "dshm") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.287086 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.287061    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.297818 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.297787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d629e2a9-e63b-4113-90ac-93363c1733ab" (UID: "d629e2a9-e63b-4113-90ac-93363c1733ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374896    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374924    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374934    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374942    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d629e2a9-e63b-4113-90ac-93363c1733ab-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374951    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374958    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkprj\" (UniqueName: \"kubernetes.io/projected/d629e2a9-e63b-4113-90ac-93363c1733ab-kube-api-access-qkprj\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374967    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d629e2a9-e63b-4113-90ac-93363c1733ab-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.374969 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.374974    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d629e2a9-e63b-4113-90ac-93363c1733ab-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.409848 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.409821    2583 generic.go:356] "Generic (PLEG): container finished" podID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerID="5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4" exitCode=0
Sep 18 20:47:55.409848 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.409845    2583 generic.go:356] "Generic (PLEG): container finished" podID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerID="04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004" exitCode=0
Sep 18 20:47:55.410007 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.409873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerDied","Data":"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"}
Sep 18 20:47:55.410007 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.409884    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"
Sep 18 20:47:55.410007 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.409917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerDied","Data":"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"}
Sep 18 20:47:55.410007 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.409934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf" event={"ID":"d629e2a9-e63b-4113-90ac-93363c1733ab","Type":"ContainerDied","Data":"26b58e2890f640031d698975e9fef2e556e968862633e80b00a0dfd59f9a0bdb"}
Sep 18 20:47:55.410007 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.409954    2583 scope.go:117] "RemoveContainer" containerID="5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"
Sep 18 20:47:55.419640 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.419617    2583 scope.go:117] "RemoveContainer" containerID="4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"
Sep 18 20:47:55.428830 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.428802    2583 scope.go:117] "RemoveContainer" containerID="04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"
Sep 18 20:47:55.433234 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.433214    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"]
Sep 18 20:47:55.437660 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.437640    2583 scope.go:117] "RemoveContainer" containerID="5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"
Sep 18 20:47:55.437987 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:47:55.437967    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4\": container with ID starting with 5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4 not found: ID does not exist" containerID="5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"
Sep 18 20:47:55.438072 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438000    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"} err="failed to get container status \"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4\": rpc error: code = NotFound desc = could not find container \"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4\": container with ID starting with 5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4 not found: ID does not exist"
Sep 18 20:47:55.438072 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438026    2583 scope.go:117] "RemoveContainer" containerID="4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"
Sep 18 20:47:55.438171 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438078    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-66858595f7-cjsvf"]
Sep 18 20:47:55.438304 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:47:55.438284    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4\": container with ID starting with 4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4 not found: ID does not exist" containerID="4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"
Sep 18 20:47:55.438349 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438315    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"} err="failed to get container status \"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4\": rpc error: code = NotFound desc = could not find container \"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4\": container with ID starting with 4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4 not found: ID does not exist"
Sep 18 20:47:55.438349 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438338    2583 scope.go:117] "RemoveContainer" containerID="04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"
Sep 18 20:47:55.438644 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:47:55.438626    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004\": container with ID starting with 04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004 not found: ID does not exist" containerID="04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"
Sep 18 20:47:55.438686 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438649    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"} err="failed to get container status \"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004\": rpc error: code = NotFound desc = could not find container \"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004\": container with ID starting with 04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004 not found: ID does not exist"
Sep 18 20:47:55.438686 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438664    2583 scope.go:117] "RemoveContainer" containerID="5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"
Sep 18 20:47:55.438912 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438892    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4"} err="failed to get container status \"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4\": rpc error: code = NotFound desc = could not find container \"5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4\": container with ID starting with 5d6549233bde5c2505f69f071a7a00c7dc00884d3c443e7e57a30ce6898a19d4 not found: ID does not exist"
Sep 18 20:47:55.438964 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.438913    2583 scope.go:117] "RemoveContainer" containerID="4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"
Sep 18 20:47:55.439147 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.439130    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4"} err="failed to get container status \"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4\": rpc error: code = NotFound desc = could not find container \"4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4\": container with ID starting with 4ec1ce17c36c627348a901ddf5bc79db8a03ba4465f0a04e5f4692321f96fcb4 not found: ID does not exist"
Sep 18 20:47:55.439192 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.439147    2583 scope.go:117] "RemoveContainer" containerID="04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"
Sep 18 20:47:55.439410 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.439375    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004"} err="failed to get container status \"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004\": rpc error: code = NotFound desc = could not find container \"04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004\": container with ID starting with 04027dbb585da1fbd3d9810597c59439c0f673edd8eb0de6a021031d6a02e004 not found: ID does not exist"
Sep 18 20:47:55.966634 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:47:55.966599    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" path="/var/lib/kubelet/pods/d629e2a9-e63b-4113-90ac-93363c1733ab/volumes"
Sep 18 20:48:00.057580 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:00.057531    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:48:10.056903 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:10.056858    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:48:20.057839 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:20.057795    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:48:28.184134 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:28.184104    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:48:28.192542 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:28.192522    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:48:29.714260 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714228    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"]
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714563    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="llm-d-routing-sidecar"
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714581    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="llm-d-routing-sidecar"
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714602    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="storage-initializer"
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714608    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="storage-initializer"
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714614    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main"
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714621    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main"
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714669    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="main"
Sep 18 20:48:29.714706 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.714677    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d629e2a9-e63b-4113-90ac-93363c1733ab" containerName="llm-d-routing-sidecar"
Sep 18 20:48:29.719071 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.719040    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.722977 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.722948    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 18 20:48:29.723219 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.723194    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-dtgsg\""
Sep 18 20:48:29.723619 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.723595    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 20:48:29.723759 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.723606    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 20:48:29.734184 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.734159    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"]
Sep 18 20:48:29.863466 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863466 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbhp\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-kube-api-access-hrbhp\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.863934 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.863762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.964757 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964658    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.964757 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.964757 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbhp\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-kube-api-access-hrbhp\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965040 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.964950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965388 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.965098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965388 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.965329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965709 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.965667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965822 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.965806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.965887 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.965856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.967805 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.967781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.967895 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.967880    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.973874 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.973832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:29.974515 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:29.974498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbhp\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-kube-api-access-hrbhp\") pod \"router-gateway-2-openshift-default-5858dff66d-7rjsl\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:30.033551 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.033517    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:30.056731 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.056700    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:48:30.308614 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.308584    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"]
Sep 18 20:48:30.311309 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:48:30.311280    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225ca7e8_6939_42c0_9d38_759f62a3bdd1.slice/crio-c80fea7f6408ee9858a680c8316c74d9f6af84de36caa723a523e75782d5e869 WatchSource:0}: Error finding container c80fea7f6408ee9858a680c8316c74d9f6af84de36caa723a523e75782d5e869: Status 404 returned error can't find the container with id c80fea7f6408ee9858a680c8316c74d9f6af84de36caa723a523e75782d5e869
Sep 18 20:48:30.313391 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.313331    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:48:30.313498 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.313455    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:48:30.313498 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.313489    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:48:30.565030 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.564940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" event={"ID":"225ca7e8-6939-42c0-9d38-759f62a3bdd1","Type":"ContainerStarted","Data":"d95d6c82c674fb5cbc52a03364c57ce0ec03bbd6afae8f5b7ec650821c9d122d"}
Sep 18 20:48:30.565030 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.565003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" event={"ID":"225ca7e8-6939-42c0-9d38-759f62a3bdd1","Type":"ContainerStarted","Data":"c80fea7f6408ee9858a680c8316c74d9f6af84de36caa723a523e75782d5e869"}
Sep 18 20:48:30.584954 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:30.584896    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podStartSLOduration=1.5848756659999998 podStartE2EDuration="1.584875666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:30.582794207 +0000 UTC m=+4204.837000815" watchObservedRunningTime="2026-09-18 20:48:30.584875666 +0000 UTC m=+4204.839082273"
Sep 18 20:48:31.034589 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:31.034555    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:31.039624 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:31.039599    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:31.570029 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:31.569992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:31.571078 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:31.571053    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:40.056847 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:40.056799    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:48:41.618341 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:41.618303    2583 generic.go:356] "Generic (PLEG): container finished" podID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerID="d95d6c82c674fb5cbc52a03364c57ce0ec03bbd6afae8f5b7ec650821c9d122d" exitCode=0
Sep 18 20:48:41.618907 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:41.618347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" event={"ID":"225ca7e8-6939-42c0-9d38-759f62a3bdd1","Type":"ContainerDied","Data":"d95d6c82c674fb5cbc52a03364c57ce0ec03bbd6afae8f5b7ec650821c9d122d"}
Sep 18 20:48:41.618907 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:41.618772    2583 scope.go:117] "RemoveContainer" containerID="d95d6c82c674fb5cbc52a03364c57ce0ec03bbd6afae8f5b7ec650821c9d122d"
Sep 18 20:48:41.619933 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:41.619897    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:48:41.620008 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:41.619987    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:48:41.620045 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:41.620034    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 20:48:42.625057 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:42.625026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" event={"ID":"225ca7e8-6939-42c0-9d38-759f62a3bdd1","Type":"ContainerStarted","Data":"72938477844f1645c0b23ec5e4c9cb0a893145a223d22b8b8646b6ddcdb62184"}
Sep 18 20:48:43.034363 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:43.034327    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:43.035428 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:43.035397    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 20:48:44.034911 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:44.034861    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 20:48:45.034303 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.034260    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 20:48:45.155379 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.154602    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"]
Sep 18 20:48:45.157613 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.157589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.161837 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.161813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 20:48:45.162236 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.162217    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 20:48:45.162663 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.162394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4lw4c\""
Sep 18 20:48:45.162929 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.162487    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:48:45.178189 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.178100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"]
Sep 18 20:48:45.194613 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.194581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-home\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.194792 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.194769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.194890 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.194825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.194957 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.194926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.195013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.194966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.195013 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.194994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjm9k\" (UniqueName: \"kubernetes.io/projected/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kube-api-access-pjm9k\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.195190 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.195026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.195190 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.195061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296276 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296276 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296276 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296276 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjm9k\" (UniqueName: \"kubernetes.io/projected/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kube-api-access-pjm9k\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296637 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296637 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296637 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-home\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.296637 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.303971 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.296974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.303971 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.297053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.303971 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.297444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-home\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.303971 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.297524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.303971 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.297591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.304373 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.304296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.305586 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.305559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.306981 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.306956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjm9k\" (UniqueName: \"kubernetes.io/projected/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kube-api-access-pjm9k\") pod \"router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.490016 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.489873    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:45.780876 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:45.780850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"]
Sep 18 20:48:45.783540 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:48:45.783499    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc001be5a_0c5a_431e_b7c2_8ce8f0a5d8c2.slice/crio-2319cfdd6e66bc18f12ea187e07168311aec032077896bc4a089203879e76dbd WatchSource:0}: Error finding container 2319cfdd6e66bc18f12ea187e07168311aec032077896bc4a089203879e76dbd: Status 404 returned error can't find the container with id 2319cfdd6e66bc18f12ea187e07168311aec032077896bc4a089203879e76dbd
Sep 18 20:48:46.035033 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:46.034935    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 20:48:46.571591 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:46.570822    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:46.645566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:46.645505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" event={"ID":"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2","Type":"ContainerStarted","Data":"49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37"}
Sep 18 20:48:46.645753 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:46.645596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" event={"ID":"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2","Type":"ContainerStarted","Data":"2319cfdd6e66bc18f12ea187e07168311aec032077896bc4a089203879e76dbd"}
Sep 18 20:48:47.034490 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:47.034449    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 20:48:48.038167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:48.038137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:48.039290 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:48.039267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:48:48.663057 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:48.663019    2583 generic.go:356] "Generic (PLEG): container finished" podID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerID="49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37" exitCode=0
Sep 18 20:48:48.663240 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:48.663099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" event={"ID":"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2","Type":"ContainerDied","Data":"49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37"}
Sep 18 20:48:49.669534 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:49.669497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" event={"ID":"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2","Type":"ContainerStarted","Data":"f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e"}
Sep 18 20:48:49.693177 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:49.693111    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podStartSLOduration=2.095450924 podStartE2EDuration="4.693090742s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.597639818s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:49.690989234 +0000 UTC m=+4223.945195839" watchObservedRunningTime="2026-09-18 20:48:49.693090742 +0000 UTC m=+4223.947297346"
Sep 18 20:48:50.057174 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:50.057118    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:48:55.490259 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:55.490222    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:55.490697 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:55.490291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:48:55.491949 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:48:55.491916    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:49:00.057136 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:00.057081    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:49:05.490965 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:05.490916    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:49:10.057074 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:10.057027    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:49:15.490809 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:15.490757    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:49:20.067373 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:20.067334    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:49:20.075261 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:20.075237    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:49:20.482173 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:20.482084    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"]
Sep 18 20:49:25.490847 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:25.490796    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:49:35.490556 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:35.490501    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:49:36.853381 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:36.853316    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" containerID="cri-o://f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e" gracePeriod=45
Sep 18 20:49:37.068274 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.068222    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 18 20:49:37.585661 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.585633    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:49:37.660055 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.659971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30be37cb-987d-4c79-8c91-798963a2885d-tls-certs\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.660055 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660027    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-tmp-dir\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.660279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-kserve-provision-location\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.660279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30be37cb-987d-4c79-8c91-798963a2885d-cabundle-cert\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.660279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660131    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-dshm\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.660279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660178    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-home\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.660279 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660226    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-model-cache\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.661585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660282    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s5gc\" (UniqueName: \"kubernetes.io/projected/30be37cb-987d-4c79-8c91-798963a2885d-kube-api-access-4s5gc\") pod \"30be37cb-987d-4c79-8c91-798963a2885d\" (UID: \"30be37cb-987d-4c79-8c91-798963a2885d\") "
Sep 18 20:49:37.661585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30be37cb-987d-4c79-8c91-798963a2885d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:49:37.661585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.660796    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-model-cache" (OuterVolumeSpecName: "model-cache") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:49:37.661585 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.661252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-home" (OuterVolumeSpecName: "home") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:49:37.663243 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.663197    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30be37cb-987d-4c79-8c91-798963a2885d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:49:37.663243 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.663207    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30be37cb-987d-4c79-8c91-798963a2885d-kube-api-access-4s5gc" (OuterVolumeSpecName: "kube-api-access-4s5gc") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "kube-api-access-4s5gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:49:37.663399 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.663313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-dshm" (OuterVolumeSpecName: "dshm") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:49:37.678894 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.678850    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:49:37.684664 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.684641    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30be37cb-987d-4c79-8c91-798963a2885d" (UID: "30be37cb-987d-4c79-8c91-798963a2885d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:49:37.761167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761144    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.761167 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761167    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.761325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761177    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5gc\" (UniqueName: \"kubernetes.io/projected/30be37cb-987d-4c79-8c91-798963a2885d-kube-api-access-4s5gc\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.761325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761185    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30be37cb-987d-4c79-8c91-798963a2885d-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.761325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761193    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.761325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761201    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.761325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761208    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30be37cb-987d-4c79-8c91-798963a2885d-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.761325 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.761215    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30be37cb-987d-4c79-8c91-798963a2885d-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:49:37.903188 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.903150    2583 generic.go:356] "Generic (PLEG): container finished" podID="30be37cb-987d-4c79-8c91-798963a2885d" containerID="f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e" exitCode=0
Sep 18 20:49:37.903188 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.903194    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"
Sep 18 20:49:37.903715 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.903221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" event={"ID":"30be37cb-987d-4c79-8c91-798963a2885d","Type":"ContainerDied","Data":"f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e"}
Sep 18 20:49:37.903715 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.903296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m" event={"ID":"30be37cb-987d-4c79-8c91-798963a2885d","Type":"ContainerDied","Data":"39ea48db6b39f782b988ef8b5dfbe46c6548f977a970123a4ba7e1f3864e4195"}
Sep 18 20:49:37.903715 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.903331    2583 scope.go:117] "RemoveContainer" containerID="f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e"
Sep 18 20:49:37.912917 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.912894    2583 scope.go:117] "RemoveContainer" containerID="84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2"
Sep 18 20:49:37.920634 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.920611    2583 scope.go:117] "RemoveContainer" containerID="f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e"
Sep 18 20:49:37.920910 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:49:37.920891    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e\": container with ID starting with f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e not found: ID does not exist" containerID="f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e"
Sep 18 20:49:37.920981 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.920919    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e"} err="failed to get container status \"f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e\": rpc error: code = NotFound desc = could not find container \"f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e\": container with ID starting with f47cc63a395c6c157dbf726968eb1e153057332b3d44b1dbd7515678bbfaf53e not found: ID does not exist"
Sep 18 20:49:37.920981 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.920935    2583 scope.go:117] "RemoveContainer" containerID="84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2"
Sep 18 20:49:37.921164 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:49:37.921150    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2\": container with ID starting with 84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2 not found: ID does not exist" containerID="84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2"
Sep 18 20:49:37.921209 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.921167    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2"} err="failed to get container status \"84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2\": rpc error: code = NotFound desc = could not find container \"84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2\": container with ID starting with 84a4c51aff07f51ecea42a2b9f3d9f8997dcd701adaab1145688250fc3553ff2 not found: ID does not exist"
Sep 18 20:49:37.927532 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.927507    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"]
Sep 18 20:49:37.931739 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.931720    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-r9z9m"]
Sep 18 20:49:37.967431 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:37.967408    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30be37cb-987d-4c79-8c91-798963a2885d" path="/var/lib/kubelet/pods/30be37cb-987d-4c79-8c91-798963a2885d/volumes"
Sep 18 20:49:45.490797 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:45.490754    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:49:55.491242 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:49:55.491195    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:50:05.490617 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:50:05.490571    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:50:15.491423 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:50:15.491338    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:50:25.490985 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:50:25.490936    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:50:35.490495 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:50:35.490450    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 20:50:45.501522 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:50:45.501486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:50:45.509204 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:50:45.509179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:51:07.293152 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.293116    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"]
Sep 18 20:51:07.293630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.293555    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main"
Sep 18 20:51:07.293630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.293571    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main"
Sep 18 20:51:07.293630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.293594    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="storage-initializer"
Sep 18 20:51:07.293630 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.293604    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="storage-initializer"
Sep 18 20:51:07.293792 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.293704    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="30be37cb-987d-4c79-8c91-798963a2885d" containerName="main"
Sep 18 20:51:07.295747 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.295725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.299402 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.299349    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 20:51:07.299535 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.299412    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 20:51:07.299535 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.299441    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-ff7kp\""
Sep 18 20:51:07.299535 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.299384    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-mjgl9\""
Sep 18 20:51:07.299714 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.299576    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 20:51:07.305786 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.305762    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"]
Sep 18 20:51:07.389304 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.389271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130b65d0-4d87-4784-b2b7-238cc25452b7-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.389486 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.389312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gqz9\" (UniqueName: \"kubernetes.io/projected/130b65d0-4d87-4784-b2b7-238cc25452b7-kube-api-access-5gqz9\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.490703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.490662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gqz9\" (UniqueName: \"kubernetes.io/projected/130b65d0-4d87-4784-b2b7-238cc25452b7-kube-api-access-5gqz9\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.490899 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.490763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130b65d0-4d87-4784-b2b7-238cc25452b7-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.493520 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.493490    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130b65d0-4d87-4784-b2b7-238cc25452b7-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.499552 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.499527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gqz9\" (UniqueName: \"kubernetes.io/projected/130b65d0-4d87-4784-b2b7-238cc25452b7-kube-api-access-5gqz9\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.607990 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.607918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:07.880397 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:07.880272    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"]
Sep 18 20:51:07.883503 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:51:07.883464    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130b65d0_4d87_4784_b2b7_238cc25452b7.slice/crio-5927e7803b19aa3589ea9fca7a0ec5fa13c33226209ad422ca6abbf5ce4013fa WatchSource:0}: Error finding container 5927e7803b19aa3589ea9fca7a0ec5fa13c33226209ad422ca6abbf5ce4013fa: Status 404 returned error can't find the container with id 5927e7803b19aa3589ea9fca7a0ec5fa13c33226209ad422ca6abbf5ce4013fa
Sep 18 20:51:08.300275 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:08.300242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" event={"ID":"130b65d0-4d87-4784-b2b7-238cc25452b7","Type":"ContainerStarted","Data":"a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c"}
Sep 18 20:51:08.300659 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:08.300289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" event={"ID":"130b65d0-4d87-4784-b2b7-238cc25452b7","Type":"ContainerStarted","Data":"5927e7803b19aa3589ea9fca7a0ec5fa13c33226209ad422ca6abbf5ce4013fa"}
Sep 18 20:51:08.300659 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:08.300319    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:51:08.317966 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:08.317921    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" podStartSLOduration=1.317907371 podStartE2EDuration="1.317907371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:51:08.315641632 +0000 UTC m=+4362.569848237" watchObservedRunningTime="2026-09-18 20:51:08.317907371 +0000 UTC m=+4362.572113974"
Sep 18 20:51:21.335122 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:21.335088    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"]
Sep 18 20:51:36.358883 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:36.358846    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main" containerID="cri-o://f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e" gracePeriod=45
Sep 18 20:51:37.108192 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.108167    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:51:37.216512 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216429    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-model-cache\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216512 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216460    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kserve-provision-location\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216512 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216478    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-dshm\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216512 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjm9k\" (UniqueName: \"kubernetes.io/projected/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kube-api-access-pjm9k\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216512 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216514    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tls-certs\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216956 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216558    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tmp-dir\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216956 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216599    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-home\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216956 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216672    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-cabundle-cert\") pod \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\" (UID: \"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2\") "
Sep 18 20:51:37.216956 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216748    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.216956 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.216880    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-model-cache\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.217460 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.217421    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:37.217652 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.217628    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-home" (OuterVolumeSpecName: "home") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.218984 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.218957    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:51:37.219084 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.219060    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kube-api-access-pjm9k" (OuterVolumeSpecName: "kube-api-access-pjm9k") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "kube-api-access-pjm9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:51:37.219145 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.219108    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-dshm" (OuterVolumeSpecName: "dshm") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.236327 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.236298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.241246 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.241223    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" (UID: "c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.317909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.317887    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tmp-dir\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.317909 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.317908    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-home\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.318038 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.317918    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-cabundle-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.318038 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.317927    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kserve-provision-location\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.318038 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.317935    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-dshm\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.318038 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.317944    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjm9k\" (UniqueName: \"kubernetes.io/projected/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-kube-api-access-pjm9k\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.318038 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.317952    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.418668 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.418605    2583 generic.go:356] "Generic (PLEG): container finished" podID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerID="f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e" exitCode=0
Sep 18 20:51:37.418668 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.418663    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"
Sep 18 20:51:37.419140 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.418697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" event={"ID":"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2","Type":"ContainerDied","Data":"f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e"}
Sep 18 20:51:37.419140 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.418797    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc" event={"ID":"c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2","Type":"ContainerDied","Data":"2319cfdd6e66bc18f12ea187e07168311aec032077896bc4a089203879e76dbd"}
Sep 18 20:51:37.419140 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.418829    2583 scope.go:117] "RemoveContainer" containerID="f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e"
Sep 18 20:51:37.427736 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.427721    2583 scope.go:117] "RemoveContainer" containerID="49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37"
Sep 18 20:51:37.435570 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.435552    2583 scope.go:117] "RemoveContainer" containerID="f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e"
Sep 18 20:51:37.435819 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:51:37.435800    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e\": container with ID starting with f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e not found: ID does not exist" containerID="f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e"
Sep 18 20:51:37.435865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.435826    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e"} err="failed to get container status \"f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e\": rpc error: code = NotFound desc = could not find container \"f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e\": container with ID starting with f1a4dc24ced09dd902b929463b82c7917b5a631731b46dc26fbefb0e1be38f5e not found: ID does not exist"
Sep 18 20:51:37.435865 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.435843    2583 scope.go:117] "RemoveContainer" containerID="49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37"
Sep 18 20:51:37.436063 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:51:37.436046    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37\": container with ID starting with 49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37 not found: ID does not exist" containerID="49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37"
Sep 18 20:51:37.436107 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.436069    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37"} err="failed to get container status \"49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37\": rpc error: code = NotFound desc = could not find container \"49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37\": container with ID starting with 49a269c78e3e27345f9d6ad71a712103be7c931a335a9bee3ee83c0c3704ad37 not found: ID does not exist"
Sep 18 20:51:37.440645 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.440624    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"]
Sep 18 20:51:37.446984 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.446965    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7cbf755cff-9xfpc"]
Sep 18 20:51:37.966365 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:37.966333    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" path="/var/lib/kubelet/pods/c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2/volumes"
Sep 18 20:51:39.306498 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:51:39.306470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:52:07.153186 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:07.153147    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"]
Sep 18 20:52:09.306754 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:09.306703    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:52:15.402265 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:15.402225    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"]
Sep 18 20:52:15.402816 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:15.402445    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy" containerID="cri-o://72938477844f1645c0b23ec5e4c9cb0a893145a223d22b8b8646b6ddcdb62184" gracePeriod=30
Sep 18 20:52:18.040854 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:18.040811    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:52:19.306796 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:19.306759    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:52:20.602835 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.602803    2583 generic.go:356] "Generic (PLEG): container finished" podID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerID="72938477844f1645c0b23ec5e4c9cb0a893145a223d22b8b8646b6ddcdb62184" exitCode=0
Sep 18 20:52:20.603220 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.602846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" event={"ID":"225ca7e8-6939-42c0-9d38-759f62a3bdd1","Type":"ContainerDied","Data":"72938477844f1645c0b23ec5e4c9cb0a893145a223d22b8b8646b6ddcdb62184"}
Sep 18 20:52:20.603220 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.602893    2583 scope.go:117] "RemoveContainer" containerID="d95d6c82c674fb5cbc52a03364c57ce0ec03bbd6afae8f5b7ec650821c9d122d"
Sep 18 20:52:20.648567 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.648544    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:52:20.741097 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-envoy\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741237 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741101    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrbhp\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-kube-api-access-hrbhp\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741237 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741137    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-certs\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741237 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741161    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-socket\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741237 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741201    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-credential-socket\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741248    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-podinfo\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741310    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istiod-ca-cert\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-token\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741477 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-data\") pod \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\" (UID: \"225ca7e8-6939-42c0-9d38-759f62a3bdd1\") "
Sep 18 20:52:20.741678 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741472    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:20.741678 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:20.741678 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741531    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:20.741840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741697    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.741840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741718    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-workload-socket\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.741840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741732    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-credential-socket\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.741840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:52:20.741840 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.741792    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-data" (OuterVolumeSpecName: "istio-data") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:20.743523 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.743500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:20.743523 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.743513    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-kube-api-access-hrbhp" (OuterVolumeSpecName: "kube-api-access-hrbhp") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "kube-api-access-hrbhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:52:20.744027 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.744013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 20:52:20.744099 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.744029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-token" (OuterVolumeSpecName: "istio-token") pod "225ca7e8-6939-42c0-9d38-759f62a3bdd1" (UID: "225ca7e8-6939-42c0-9d38-759f62a3bdd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:52:20.842793 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.842755    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-podinfo\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.842793 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.842789    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istiod-ca-cert\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.842793 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.842798    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-token\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.843003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.842807    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-data\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.843003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.842815    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225ca7e8-6939-42c0-9d38-759f62a3bdd1-istio-envoy\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:20.843003 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:20.842822    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbhp\" (UniqueName: \"kubernetes.io/projected/225ca7e8-6939-42c0-9d38-759f62a3bdd1-kube-api-access-hrbhp\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:21.607996 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:21.607964    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"
Sep 18 20:52:21.608525 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:21.607964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl" event={"ID":"225ca7e8-6939-42c0-9d38-759f62a3bdd1","Type":"ContainerDied","Data":"c80fea7f6408ee9858a680c8316c74d9f6af84de36caa723a523e75782d5e869"}
Sep 18 20:52:21.608525 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:21.608103    2583 scope.go:117] "RemoveContainer" containerID="72938477844f1645c0b23ec5e4c9cb0a893145a223d22b8b8646b6ddcdb62184"
Sep 18 20:52:21.631604 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:21.631581    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"]
Sep 18 20:52:21.639656 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:21.639635    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-7rjsl"]
Sep 18 20:52:21.966165 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:21.966082    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" path="/var/lib/kubelet/pods/225ca7e8-6939-42c0-9d38-759f62a3bdd1/volumes"
Sep 18 20:52:22.173488 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:22.173424    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main" containerID="cri-o://a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c" gracePeriod=45
Sep 18 20:52:29.306095 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:29.306053    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:52:29.306483 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:29.306164    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:52:39.305882 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:39.305846    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:52:49.306328 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:49.306288    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:52:52.427822 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.427794    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:52:52.465825 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.465758    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gqz9\" (UniqueName: \"kubernetes.io/projected/130b65d0-4d87-4784-b2b7-238cc25452b7-kube-api-access-5gqz9\") pod \"130b65d0-4d87-4784-b2b7-238cc25452b7\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") "
Sep 18 20:52:52.465825 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.465825    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130b65d0-4d87-4784-b2b7-238cc25452b7-tls-certs\") pod \"130b65d0-4d87-4784-b2b7-238cc25452b7\" (UID: \"130b65d0-4d87-4784-b2b7-238cc25452b7\") "
Sep 18 20:52:52.468144 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.468117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130b65d0-4d87-4784-b2b7-238cc25452b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "130b65d0-4d87-4784-b2b7-238cc25452b7" (UID: "130b65d0-4d87-4784-b2b7-238cc25452b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:52:52.468272 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.468183    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130b65d0-4d87-4784-b2b7-238cc25452b7-kube-api-access-5gqz9" (OuterVolumeSpecName: "kube-api-access-5gqz9") pod "130b65d0-4d87-4784-b2b7-238cc25452b7" (UID: "130b65d0-4d87-4784-b2b7-238cc25452b7"). InnerVolumeSpecName "kube-api-access-5gqz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:52:52.567071 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.567043    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gqz9\" (UniqueName: \"kubernetes.io/projected/130b65d0-4d87-4784-b2b7-238cc25452b7-kube-api-access-5gqz9\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:52.567071 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.567070    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130b65d0-4d87-4784-b2b7-238cc25452b7-tls-certs\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:52:52.736566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.736486    2583 generic.go:356] "Generic (PLEG): container finished" podID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerID="a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c" exitCode=0
Sep 18 20:52:52.736566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.736528    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"
Sep 18 20:52:52.736566 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.736549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" event={"ID":"130b65d0-4d87-4784-b2b7-238cc25452b7","Type":"ContainerDied","Data":"a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c"}
Sep 18 20:52:52.736775 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.736593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl" event={"ID":"130b65d0-4d87-4784-b2b7-238cc25452b7","Type":"ContainerDied","Data":"5927e7803b19aa3589ea9fca7a0ec5fa13c33226209ad422ca6abbf5ce4013fa"}
Sep 18 20:52:52.736775 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.736616    2583 scope.go:117] "RemoveContainer" containerID="a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c"
Sep 18 20:52:52.745191 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.745175    2583 scope.go:117] "RemoveContainer" containerID="a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c"
Sep 18 20:52:52.745451 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:52:52.745435    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c\": container with ID starting with a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c not found: ID does not exist" containerID="a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c"
Sep 18 20:52:52.745504 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.745459    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c"} err="failed to get container status \"a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c\": rpc error: code = NotFound desc = could not find container \"a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c\": container with ID starting with a803d28c34a0c1e97572de28c2654001f73f858b5808d24d8508277b9d6e149c not found: ID does not exist"
Sep 18 20:52:52.758506 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.758485    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"]
Sep 18 20:52:52.767498 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:52.767475    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-khzdl"]
Sep 18 20:52:53.965905 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:53.965872    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" path="/var/lib/kubelet/pods/130b65d0-4d87-4784-b2b7-238cc25452b7/volumes"
Sep 18 20:52:54.990505 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990473    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzczt/must-gather-77gps"]
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990736    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990748    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990762    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990767    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990775    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="storage-initializer"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990781    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="storage-initializer"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990792    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990798    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990804    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990809    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990851    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990860    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c001be5a-0c5a-431e-b7c2-8ce8f0a5d8c2" containerName="main"
Sep 18 20:52:54.990937 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990867    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="130b65d0-4d87-4784-b2b7-238cc25452b7" containerName="main"
Sep 18 20:52:54.991378 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.990960    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="225ca7e8-6939-42c0-9d38-759f62a3bdd1" containerName="istio-proxy"
Sep 18 20:52:54.995146 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.995125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:54.997780 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.997758    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzczt\"/\"kube-root-ca.crt\""
Sep 18 20:52:54.997874 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.997776    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zzczt\"/\"default-dockercfg-q9h9j\""
Sep 18 20:52:54.997874 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:54.997787    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzczt\"/\"openshift-service-ca.crt\""
Sep 18 20:52:55.005817 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.005796    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzczt/must-gather-77gps"]
Sep 18 20:52:55.086186 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.086158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57p2d\" (UniqueName: \"kubernetes.io/projected/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-kube-api-access-57p2d\") pod \"must-gather-77gps\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") " pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:55.086314 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.086194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-must-gather-output\") pod \"must-gather-77gps\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") " pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:55.186598 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.186567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57p2d\" (UniqueName: \"kubernetes.io/projected/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-kube-api-access-57p2d\") pod \"must-gather-77gps\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") " pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:55.186730 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.186613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-must-gather-output\") pod \"must-gather-77gps\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") " pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:55.186987 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.186967    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-must-gather-output\") pod \"must-gather-77gps\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") " pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:55.195834 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.195813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57p2d\" (UniqueName: \"kubernetes.io/projected/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-kube-api-access-57p2d\") pod \"must-gather-77gps\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") " pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:55.304658 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.304632    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:52:55.578834 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.578799    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzczt/must-gather-77gps"]
Sep 18 20:52:55.581567 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:52:55.581541    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabfe64ee_e7fd_4cc5_ad05_7cfd79c3f26c.slice/crio-7d28e9973d61b4329e388025243354de2062a9c0a979f65da7b7f664b35a41ec WatchSource:0}: Error finding container 7d28e9973d61b4329e388025243354de2062a9c0a979f65da7b7f664b35a41ec: Status 404 returned error can't find the container with id 7d28e9973d61b4329e388025243354de2062a9c0a979f65da7b7f664b35a41ec
Sep 18 20:52:55.583762 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.583747    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:52:55.750789 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:52:55.750747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzczt/must-gather-77gps" event={"ID":"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c","Type":"ContainerStarted","Data":"7d28e9973d61b4329e388025243354de2062a9c0a979f65da7b7f664b35a41ec"}
Sep 18 20:53:02.788719 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:02.788676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzczt/must-gather-77gps" event={"ID":"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c","Type":"ContainerStarted","Data":"2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1"}
Sep 18 20:53:02.789148 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:02.788732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzczt/must-gather-77gps" event={"ID":"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c","Type":"ContainerStarted","Data":"672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546"}
Sep 18 20:53:19.591085 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:19.591052    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-sl7nz_ea16aa43-9fea-4f01-821f-c3fe02e87a7e/manager/0.log"
Sep 18 20:53:19.600222 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:19.600197    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tw5h6_52157160-3c1d-4d59-88f9-2ea8f2e1e285/limitador/0.log"
Sep 18 20:53:20.869723 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:20.869690    2583 generic.go:356] "Generic (PLEG): container finished" podID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerID="672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546" exitCode=0
Sep 18 20:53:20.870148 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:20.869763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzczt/must-gather-77gps" event={"ID":"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c","Type":"ContainerDied","Data":"672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546"}
Sep 18 20:53:20.870148 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:20.870082    2583 scope.go:117] "RemoveContainer" containerID="672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546"
Sep 18 20:53:21.796646 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:21.796614    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzczt_must-gather-77gps_abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c/gather/0.log"
Sep 18 20:53:25.317314 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:25.317279    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pkqrg_4fd05598-f32e-42b9-82a7-cb695c62d50c/global-pull-secret-syncer/0.log"
Sep 18 20:53:25.480246 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:25.480216    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n9s2s_3356ecda-7f08-4f66-a08e-2cae194cc7a0/konnectivity-agent/0.log"
Sep 18 20:53:25.539687 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:25.539656    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-25.ec2.internal_3fba69533f2373986d0e5941a0f2828d/haproxy/0.log"
Sep 18 20:53:27.238437 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.238400    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zzczt/must-gather-77gps"]
Sep 18 20:53:27.238858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.238576    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-zzczt/must-gather-77gps" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerName="copy" containerID="cri-o://2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1" gracePeriod=2
Sep 18 20:53:27.242236 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.242201    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zzczt/must-gather-77gps"]
Sep 18 20:53:27.464700 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.464676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzczt_must-gather-77gps_abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c/copy/0.log"
Sep 18 20:53:27.466322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.466303    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:53:27.468600 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:53:27.468577    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-77gps\" is forbidden: User \"system:node:ip-10-0-133-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzczt\": no relationship found between node 'ip-10-0-133-25.ec2.internal' and this object" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:53:27.557484 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.557459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-must-gather-output\") pod \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") "
Sep 18 20:53:27.557617 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.557498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57p2d\" (UniqueName: \"kubernetes.io/projected/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-kube-api-access-57p2d\") pod \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\" (UID: \"abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c\") "
Sep 18 20:53:27.559685 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.559661    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-kube-api-access-57p2d" (OuterVolumeSpecName: "kube-api-access-57p2d") pod "abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" (UID: "abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c"). InnerVolumeSpecName "kube-api-access-57p2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:53:27.564380 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.564339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" (UID: "abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:53:27.658858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.658836    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-must-gather-output\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:53:27.658858 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.658857    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57p2d\" (UniqueName: \"kubernetes.io/projected/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c-kube-api-access-57p2d\") on node \"ip-10-0-133-25.ec2.internal\" DevicePath \"\""
Sep 18 20:53:27.902938 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.902864    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzczt_must-gather-77gps_abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c/copy/0.log"
Sep 18 20:53:27.903254 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.903232    2583 generic.go:356] "Generic (PLEG): container finished" podID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerID="2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1" exitCode=143
Sep 18 20:53:27.903322 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.903273    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:53:27.903385 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.903331    2583 scope.go:117] "RemoveContainer" containerID="2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1"
Sep 18 20:53:27.905654 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:53:27.905630    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-77gps\" is forbidden: User \"system:node:ip-10-0-133-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzczt\": no relationship found between node 'ip-10-0-133-25.ec2.internal' and this object" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:53:27.911824 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.911807    2583 scope.go:117] "RemoveContainer" containerID="672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546"
Sep 18 20:53:27.913917 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:53:27.913897    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-77gps\" is forbidden: User \"system:node:ip-10-0-133-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzczt\": no relationship found between node 'ip-10-0-133-25.ec2.internal' and this object" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:53:27.923843 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.923829    2583 scope.go:117] "RemoveContainer" containerID="2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1"
Sep 18 20:53:27.924091 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:53:27.924073    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1\": container with ID starting with 2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1 not found: ID does not exist" containerID="2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1"
Sep 18 20:53:27.924159 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.924099    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1"} err="failed to get container status \"2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1\": rpc error: code = NotFound desc = could not find container \"2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1\": container with ID starting with 2435ee2392384365e7a2cc7a3b34ca96caafb6df1e70e5d920310a90c600ebe1 not found: ID does not exist"
Sep 18 20:53:27.924159 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.924116    2583 scope.go:117] "RemoveContainer" containerID="672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546"
Sep 18 20:53:27.924385 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:53:27.924347    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546\": container with ID starting with 672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546 not found: ID does not exist" containerID="672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546"
Sep 18 20:53:27.924439 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.924393    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546"} err="failed to get container status \"672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546\": rpc error: code = NotFound desc = could not find container \"672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546\": container with ID starting with 672160c374acb07533586eadb4ab3290adebaaef330f9a42ef45b9ea8ca79546 not found: ID does not exist"
Sep 18 20:53:27.965398 ip-10-0-133-25 kubenswrapper[2583]: E0918 20:53:27.965374    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-77gps\" is forbidden: User \"system:node:ip-10-0-133-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzczt\": no relationship found between node 'ip-10-0-133-25.ec2.internal' and this object" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" pod="openshift-must-gather-zzczt/must-gather-77gps"
Sep 18 20:53:27.966559 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:27.966540    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" path="/var/lib/kubelet/pods/abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c/volumes"
Sep 18 20:53:28.210427 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:28.210333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:53:28.217242 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:28.217217    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzl5s_116348dd-64ad-47d4-a5a8-b41540fd7a67/ovn-acl-logging/0.log"
Sep 18 20:53:30.967519 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:30.967488    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-sl7nz_ea16aa43-9fea-4f01-821f-c3fe02e87a7e/manager/0.log"
Sep 18 20:53:30.989465 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:30.989443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tw5h6_52157160-3c1d-4d59-88f9-2ea8f2e1e285/limitador/0.log"
Sep 18 20:53:32.897292 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:32.897214    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjrfz_2f950c7c-633a-4ad7-83cf-2f2f50dc7e39/node-exporter/0.log"
Sep 18 20:53:32.919668 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:32.919640    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjrfz_2f950c7c-633a-4ad7-83cf-2f2f50dc7e39/kube-rbac-proxy/0.log"
Sep 18 20:53:32.948854 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:32.948830    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjrfz_2f950c7c-633a-4ad7-83cf-2f2f50dc7e39/init-textfile/0.log"
Sep 18 20:53:33.088634 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:33.088606    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cf24ca2-cc03-414b-8506-6657d00c7b3a/prometheus/0.log"
Sep 18 20:53:33.107178 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:33.107151    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cf24ca2-cc03-414b-8506-6657d00c7b3a/config-reloader/0.log"
Sep 18 20:53:33.132702 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:33.132666    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cf24ca2-cc03-414b-8506-6657d00c7b3a/thanos-sidecar/0.log"
Sep 18 20:53:33.153758 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:33.153672    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cf24ca2-cc03-414b-8506-6657d00c7b3a/kube-rbac-proxy-web/0.log"
Sep 18 20:53:33.179074 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:33.179048    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cf24ca2-cc03-414b-8506-6657d00c7b3a/kube-rbac-proxy/0.log"
Sep 18 20:53:33.203052 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:33.203020    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cf24ca2-cc03-414b-8506-6657d00c7b3a/kube-rbac-proxy-thanos/0.log"
Sep 18 20:53:33.229045 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:33.229021    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2cf24ca2-cc03-414b-8506-6657d00c7b3a/init-config-reloader/0.log"
Sep 18 20:53:35.318618 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.318587    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"]
Sep 18 20:53:35.318999 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.318828    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerName="gather"
Sep 18 20:53:35.318999 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.318838    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerName="gather"
Sep 18 20:53:35.318999 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.318856    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerName="copy"
Sep 18 20:53:35.318999 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.318861    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerName="copy"
Sep 18 20:53:35.318999 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.318908    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerName="copy"
Sep 18 20:53:35.318999 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.318919    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfe64ee-e7fd-4cc5-ad05-7cfd79c3f26c" containerName="gather"
Sep 18 20:53:35.322848 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.322828    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.325595 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.325576    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcrhg\"/\"kube-root-ca.crt\""
Sep 18 20:53:35.325712 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.325600    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcrhg\"/\"openshift-service-ca.crt\""
Sep 18 20:53:35.326703 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.326683    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kcrhg\"/\"default-dockercfg-2sg28\""
Sep 18 20:53:35.336019 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.335999    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"]
Sep 18 20:53:35.416099 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.416079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-proc\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.416208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.416120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-lib-modules\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.416208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.416139    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-podres\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.416208 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.416190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-sys\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.416340 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.416219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682hr\" (UniqueName: \"kubernetes.io/projected/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-kube-api-access-682hr\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.516675 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682hr\" (UniqueName: \"kubernetes.io/projected/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-kube-api-access-682hr\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.516783 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-proc\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.516783 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-lib-modules\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.516884 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-proc\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.516884 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-podres\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.516884 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-sys\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.516884 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-lib-modules\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.517032 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-podres\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.517032 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.516943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-sys\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.525123 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.525101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-682hr\" (UniqueName: \"kubernetes.io/projected/b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d-kube-api-access-682hr\") pod \"perf-node-gather-daemonset-g62xc\" (UID: \"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.632412 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.632335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:35.900817 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.900741    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"]
Sep 18 20:53:35.904097 ip-10-0-133-25 kubenswrapper[2583]: W0918 20:53:35.904067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb3a1cb7c_be93_48f9_a0fa_ef5bd4c58c2d.slice/crio-13c541479fc146297ca3ac228a6b80449d381c992a5752d3eb0e88bd4da7e7be WatchSource:0}: Error finding container 13c541479fc146297ca3ac228a6b80449d381c992a5752d3eb0e88bd4da7e7be: Status 404 returned error can't find the container with id 13c541479fc146297ca3ac228a6b80449d381c992a5752d3eb0e88bd4da7e7be
Sep 18 20:53:35.937825 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:35.937712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc" event={"ID":"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d","Type":"ContainerStarted","Data":"13c541479fc146297ca3ac228a6b80449d381c992a5752d3eb0e88bd4da7e7be"}
Sep 18 20:53:36.943387 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:36.943330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc" event={"ID":"b3a1cb7c-be93-48f9-a0fa-ef5bd4c58c2d","Type":"ContainerStarted","Data":"d6ed261535635d4ec7d4a19256bc5fa5578016f64244b14469b8c1a6bf0a1616"}
Sep 18 20:53:36.943771 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:36.943524    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:36.960975 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:36.960930    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc" podStartSLOduration=1.960918397 podStartE2EDuration="1.960918397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:53:36.959988851 +0000 UTC m=+4511.214195455" watchObservedRunningTime="2026-09-18 20:53:36.960918397 +0000 UTC m=+4511.215125001"
Sep 18 20:53:37.356041 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:37.356011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wbtwc_1044dfff-ccf8-4561-b87c-c5fc588f6603/dns/0.log"
Sep 18 20:53:37.376486 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:37.376458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wbtwc_1044dfff-ccf8-4561-b87c-c5fc588f6603/kube-rbac-proxy/0.log"
Sep 18 20:53:37.423978 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:37.423954    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spvxq_077e0f30-ad3e-44a5-bf8d-4cf88903e1b0/dns-node-resolver/0.log"
Sep 18 20:53:38.069999 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:38.069939    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fx5vm_a3f524e7-7958-47df-8762-0839d84b63fd/node-ca/0.log"
Sep 18 20:53:39.916449 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:39.916406    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shv9m_2e30c6ab-20fd-40dd-952a-9dfe7562d4cb/serve-healthcheck-canary/0.log"
Sep 18 20:53:40.382541 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:40.382508    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dng7g_30e92495-9fea-41e7-8ec2-fdfb24825356/kube-rbac-proxy/0.log"
Sep 18 20:53:40.403818 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:40.403791    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dng7g_30e92495-9fea-41e7-8ec2-fdfb24825356/exporter/0.log"
Sep 18 20:53:40.427988 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:40.427964    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dng7g_30e92495-9fea-41e7-8ec2-fdfb24825356/extractor/0.log"
Sep 18 20:53:42.957144 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:42.957116    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-g62xc"
Sep 18 20:53:43.641764 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:43.641736    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d69b774b8-9pb2m_b8cab9b4-b072-42f9-9c57-1986f36fd954/manager/0.log"
Sep 18 20:53:46.969824 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:46.969785    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-j8fxj_eea59772-56aa-4043-a6a3-e75d99fb3162/s3-init/0.log"
Sep 18 20:53:46.987265 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:46.987241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-j8fxj_eea59772-56aa-4043-a6a3-e75d99fb3162/download-hf-model/0.log"
Sep 18 20:53:51.530127 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:51.530094    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-wsbmd_1e14754c-369c-43a1-86c2-c89a4594a94a/migrator/0.log"
Sep 18 20:53:51.549308 ip-10-0-133-25 kubenswrapper[2583]: I0918 20:53:51.549280    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-wsbmd_1e14754c-369c-43a1-86c2-c89a4594a94a/graceful-termination/0.log"